PHP: Best way to do a big data cron job? [closed]

  mysql, php

I have an API endpoint that responds with an array of IDs given a unique username as a parameter. I have over 1000 unique usernames in a MYSQL database (the response array from the API is just random numbers chosen by the users). Response from API below:

{3, 1, 4, 7, 5}

So, I take the response given 1 of the usernames as a parameter to the API, parse it and take each ID and check it against the local database to see if it’s there, if it’s not, I insert in into the database and send a request to another endpoint (a push notification endpoint), if it is in the database, I do nothing.

The thing is, I have to get all the usernames (all 1000 of them) and take the response of each username and parse it and take each ID and check it against the local database to see if it’s there, if it’s not, I insert in into the database and send a request to another endpoint (a push notification endpoint), if it is in the database, I do nothing.

It’s a hefty request. I estimate about a 10 min request or even more for a cronjob that has to be done every 5 mins. I don’t have access to the endpoint’s backend to tie my backend to it to make my life easier. The only way I can access anything is through this endpoint.

How can I process each username through the endpoint in one go without it being a hefty request? I cannot break up the workload into parts because all users have to get notifications at the same time or close to the same time.

P.S. This is an alternative example to a bigger problem due to not leaking any info of the company I work for.

Source: Ask PHP

LEAVE A COMMENT