Saving to a new file name and/or type will keep the old file open instead of new file

If anyone is looking at the accepted answer and thinking "that doesn't solve my problem" do check your keyboard shortcuts. In earlier versions, the shortcut Shift + Ctrl + S used to map to Save As... but it now maps to Save a Copy...

"Save As" is the option you need if you want the new file to be the one that remains open in the program. If you're used to using this shortcut and can't understand why Photoshop suddenly decided to keep the old file open instead of the new one, you can go to Edit > Keyboard Shortcuts... to change it.

If I can understad what are you talking about then behaviour is the same in newest Photoshop 21.0.3 as I am testing. It will not change to test.jpg if you have layered document in Photoshop, do you understand why? Because in that case it must flatten your document since saved document in JPEG file format does not have any layers. Which verson of Photoshop and OS you are using?

Also if you are editing an opened fat jpeg and add layers then use Save.  Photoshop will switch to Save As to give you the opportunity to save a layered document.  If you use save as to save a jpeg the file saved will be a flat jpeg file. The open document in Photoshop will still be layered though there is no layered backing file

Flatten the document in Photoshop  and save a flat file Photoshop will switch to the flat document in Photoshop it will look like a rename of the document in Photoshop.

There also seems be as behavior exception. If you always flatten  first and then save a flat files. Photoshop will save the Flat document and change the document in Photoshop to the newly saved document. Both files will be on your disk. However. Photoshop will switch between documents and Photoshop 2020 version 21.1.0 still seems to work that way  only one document will be open in photoshop. Once a layer document is saved. If you save a flat file from a document with  a layered backing file even if you first flatten the flat file will be saved but it will not be open in Photoshop. Photoshop behaves strangely at times. Though it look like its rename there are two images files that will have diffent document manes in Photoshop.

You are wrong.  If you were working on a new document that was unsaved  the document may have any document name like ImageName.acr extension.  When You save the Document.  The Document would then have a backing file and Photoshop would change the open document name in Photoshop  to something like ImageName.saved file type ext.  That Document name is then not changeable.  If you then do a "Save As" different image File Format.  The document open in Photoshop Document Name will not change nor will its backing file.  There will be a new files on your system DocName.ext.  You can open that as a document and also have it open in Photoshop.  Its name will not be changeable document names are read only when there is a backing file.  The only way a Script can changed that document name it to close the document rename the documents backing file then open the backing file to get the new name open in Photoshop.  Here is a script you can use to renames a open document. It will not rename a new document that has not been saved.  It will not save the current document if it needs to be saved for unsaved changes but close will give you the option to save. The Script will not changes the documents file type extension. The Script will not check the Name you entered is a valid file name.  If you enter an invalid file name I expect the file rename will fail.   RenameDocument.jsx

#target photoshop
if(documents.length > 0){
	try{
		var tmp = app.activeDocument.fullName.name;
		var newName = "";
		while (newName=="") { newName=prompt("Enter Document Name",newName);}
		renameDoc(newName);
	}
	catch(e){alert("Document has no backing File");}
}
function renameDoc(fName){
	try{
		var originalDialogMode = app.displayDialogs;
		app.displayDialogs = DialogModes.ERROR;
		var docRef = activeDocument;
		var path = docRef.path  + "/";		
		var psFile = File(docRef.fullName);
		var psFileName = psFile.name.split('.');
		if(psFileName.length > 1){
			fName = fName + '.' + psFileName[psFileName.length - 1];
		}
		var path = docRef.path  + "/";
		docRef.close();
		psFile.rename(fName);
		open(File(path + fName));
		app.displayDialogs = originalDialogMode;
	}
	catch(e){
		alert(e + e.line);
		app.displayDialogs = originalDialogMode;
	}
}