I am currently working on an API that downloads a file from a server and writes the content to a specified location using HttpURLConnection. The code snippet I am using for the file transfer is as follows:
HttpURLConnection urlConnection = (HttpURLConnection) requestUrl.openConnection();
try (BufferedOutputStream fileOutputStream = new BufferedOutputStream(new FileOutputStream(filePath, true))) {
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = bufferedInputStream.read(buffer)) != -1) {
fileOutputStream.write(buffer, 0, bytesRead);
}
}
The current implementation works fine, but I need to show the progress of the file transfer in percentage, especially since the files can be quite large and the transfer may take a long time.
I tried to develop new api. But I am not getting way to handle for the scenario to make is user specific means there may be multiple users transferring the file simultaneously. So, I want to show the progress of the particular file which is being transferred by particular user.`
Any guidance or code examples would be greatly appreciated!
I am currently working on an API that downloads a file from a server and writes the content to a specified location using HttpURLConnection. The code snippet I am using for the file transfer is as follows:
HttpURLConnection urlConnection = (HttpURLConnection) requestUrl.openConnection();
try (BufferedOutputStream fileOutputStream = new BufferedOutputStream(new FileOutputStream(filePath, true))) {
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = bufferedInputStream.read(buffer)) != -1) {
fileOutputStream.write(buffer, 0, bytesRead);
}
}
The current implementation works fine, but I need to show the progress of the file transfer in percentage, especially since the files can be quite large and the transfer may take a long time.
I tried to develop new api. But I am not getting way to handle for the scenario to make is user specific means there may be multiple users transferring the file simultaneously. So, I want to show the progress of the particular file which is being transferred by particular user.`
Any guidance or code examples would be greatly appreciated!
So to calculate the percentage you'd have something like this:
@Autowired
private ProgressRepository progress;
private ExecutorService executorService = ExecutorService.newFixedThreadPool(5);
...
public Integer performUploadJob( URL requestUrl ) {
HttpURLConnection urlConnection = (HttpURLConnection) requestUrl.openConnection();
final long total = urlConnection.getContentLength();
int progressId = progress.startProgress( total );
// this executes this inside the web tier on a background thread!!
executorService.submit( () => {
try (BufferedOutputStream fileOutputStream = new BufferedOutputStream(new FileOutputStream(filePath, true))) {
byte[] buffer = new byte[8 * 1024];
int bytesRead;
long progress = 0L;
long lastUpdatedOn = System.currentTimeInMillis();
while ((bytesRead = bufferedInputStream.read(buffer)) != -1) {
fileOutputStream.write(buffer, 0, bytesRead);
progress += bytesRead;
long now = System.currentTimeInMillis();
if( now - lastUpdatedOn > 5 * 1000L ) {
// only update progress every 5 seconds.
// This is a trade off between going faster vs
// updating progress. If you updated every 8k
// bytes it'll dramatically increase the time
// needed to upload.
progress.updateProgress( progress );
lastUpdatedOn = now
}
}
// just in case we had some left over
progress.updateProgress( progress );
}
});
return progressId;
}
So progress / total will give you a percentage, but I don't think you need to calculate that in your Java code. Just the total and progress are enough. Your front end can make it pretty with a percentage just as easily (you mentioned a front-end, but didn't clarify if that was a web-based application). I'm assuming you are doing this in a web-based app since you also said something about multiple users.
The issue of several connections working on downloads would need to be addressed by storing progress and total in your DB. So I'd have a table like the following:
create table UploadProgress(
id int not null AUTO_INCREMENT,
progress BIGINT not null default 0,
total BIGINT not null
created_on datetime CURRENT_TIMESTAMP
)
Then add the methods to insert and update. The id field can be used by the front-end to query the state of the upload. And your server can return the id field to the front-end prior to triggering your download. You'll have to move your download code off the request thread to do so which means you need a thread pool to do that (ie ExecutorService). This would effectively support a polling model. It's a simple solution...
Update: I've fleshed out more of what I mean by the simple solution. This is a mostly all Java solution (no extra dependencies). And it runs the upload on a background thread through the ExecutorService. I DID NOT address how errors were handled but modifying the Progress DB table can store errors too. Just be aware of the following architecture issues:
However, running a thread on the web server, while it works, does have impacts for bigger picture considerations. If you deploy updates in the middle of someone downloading this file it'll kill downloads. So it's best to consider moving this off into a separate process that runs in the background. That will separate the web tier from this backend work that can run without interruption should you deploy code.
Update: If you are using Spring/Springboot you could look at JobRunr (https://www.baeldung/java-jobrunr-spring) to accomplish what my suggestion is too.
My suggestion is to use message queuing system (JMS, AWS SNS, ActiveMQ, RabbitMQ, etc) to broadcast messages to this backend when starting downloads. If you do this then you can also broadcast the progress back to the web tier. You could still use polling model with the message queuing model, but...
This would allow you to shift to using Websockets where you relay those messages to your front-end directly from the backend. This allows you to skip the DB persistence I talked about under the polling model if you wished (you can still use it if you want). That'll save you having to manage old useless records in a DB for progress (remember the created_on date? That's what that was for). Of course this solution has more infrastructure and deployment considerations to undertake. I would go with this model if I had other background operations I needed to support as that would justify the added complexity of deployment and architecture.
HttpURLConnection. That said, have a look at something like tongfei.me/progressbar. It's not really clear though how you want to display the progress. Also, if you're using Jakarta REST (JAX-RS), there are ways to download the file without using a connection like that. – James R. Perkins Commented Mar 24 at 14:26