In addition to his other work, General Groves explains in Now It Can Be Told: The Story of the Manhattan Project, Arthur Compton had been assigned over-all responsibility for the physics of bomb development:
As a first step in June, 1942, he had appointed Dr. J. Robert Oppenheimer to take charge of this particular phase of the project. Oppenheimer was then at the University of California at Berkeley. He began work on the problem with a small group of theoretical physicists.
[…]
Adding to my cause for doubt, no one with whom I talked showed any great enthusiasm about Oppenheimer as a possible director of the project.
My own feeling was that he was well qualified to handle the theoretical aspects of the work, but how he would do on the practical experimentation, or how he would handle the administrative responsibilities, I had no idea.
[…]
Of the men within our organization I had no doubt that Ernest Lawrence could handle it. He was an outstanding experimental physicist, and this was a job for an experimental physicist. However, he could not be spared from his work on the electromagnetic process; in fact, without him we would have had to drop it, for it was far too difficult and complex for anyone else. I knew of no one then and I know of no one now, besides Ernest Lawrence, who could unquestionably have carried that development through to a successful conclusion.
[…]
Oppenheimer had two major disadvantages — he had had almost no administrative experience of any kind, and he was not a Nobel Prize winner. Because of the latter lack, he did not then have the prestige among his fellow scientists that I would have liked the project leader to possess. The heads of our three major laboratories — Lawrence at Berkeley, Urey at Columbia, and Compton at Chicago — were all Nobel Prize winners, and Compton had several Nobel Prize winners working under him. There was a strong feeling among most of the scientific people with whom I discussed this matter that the head of Project Y should also be one.
[…]
His background included much that was not to our liking by any means. The security organization, which was not yet under my complete control, was unwilling to clear him because of certain of his associations, particularly in the past. I was thoroughly familiar with everything that had been reported about Oppenheimer. As always in security matters of such importance, I had read all the available original evidence; I did not depend upon the conclusions of the security officers.
Finally, because I felt that his potential value outweighed any security risk, and to remove the matter from further discussion, I personally wrote and signed the following instructions to the District Engineer on July 20, 1943:
In accordance with my verbal directions of July 15, it is desired that clearance be issued for the employment of Julius Robert Oppenheimer without delay, irrespective of the information which you have concerning Mr. Oppenheimer. He is absolutely essential to the project.