James limped out of Tuesday's Champions League win at AC Milan with a knee problem and sources have told ESPN that there are growing fears the 22-year-old may require an operation.
"He's due to see a specialist over the weekend, so until we get that information there's not too much else I can add unfortunately," Potter said, speaking at a news conference on Friday.
"The rest would be speculation. We will wait and see. We need to check the discomfort and see a specialist.
- Notebook: The inside stories from world football
"Yesterday he felt not too bad, but until you get these things checked you never know. We will wait and see. There's no point looking into the doomsday scenario yet, we need to check and then go forward."
A decision is expected to be taken this weekend and, should he require surgery, James would almost certainly be ruled out of England's World Cup squad.
That scenario would be a huge blow to manager Gareth Southgate, who is already sweating over the availability of Manchester City's Kyle Walker following a groin operation while Liverpool's Trent Alexander-Arnold is sidelined with an ankle problem.
The France international, who has struggled with repeated muscular, hamstring and knee problems, will also visit a specialist having failed to make an appearance since Aug. 14.
"Again, he's due to see a consultant at the weekend," Potter added. "It's a setback so it's not good news but, anything other than that at this stage, I can't give you anything more.
"Historically there are things which have happened which is a concern, so we need to get to the bottom of that.
"It's a setback at this stage of this rehab which is worrying, but all we can do is get the right diagnosis, get the right consultation, and try and help him get back to full fitness and enjoying his football."