Serve both raw data and populated documents to Front end

While working at REDACTED, I reported the following issue.

• Back end programmers tend to provide raw data only.
• Front end programmers can retrieve data from many endpoints.

Shouldn’t we then use the browser to aggregate data?

No, because if there is a reason for having interfaces between different systems it is to hide their inner workings and offer more coherent, more abstract, more complete and, in general, easier to understand operations and results.

The Front end side needs both aggregated and raw data fron the Back end.

For example, take a search results page for authors’ profiles based on their language.

1. A language select box needs to show English, Spanish, … (raw data) for the user to search one
2. The resulting profiles need each to show the number of authored documents, and their titles (populated documents)

Of course I could use three normalized endpoints:

1. getLanguages() for the options in 1.
2. getProfiles(page) for the results in 2.
3. getDocuments(authors) for the titles in 2.

but that would

• add a call from Front end to Back end, causing unnecessary slowness for the end user;
• add knowledge to Front end about Back end implementation details;
• add complications to Front end, just because.

Those complications are:

1. get the authors from the profiles in the current page;
2. asynchronously get the documents from getDocuments(authors), taking care of possible errors;
3. then group the titles and the count of those documents by author;
4. then make those additional data appear in matching rows on the page.

Could the operations above be performed in the Back end? Not only they could, but they would be easier and faster there, because getDocuments would not be asynchronous.

Instead, a Back end engineer told me he wouldn’t populate a response because it would set a precedent and all other Front end engineers would ask the same at any later time. Another Back end engineer said that long ago they had foreseen the need for a data aggregator but lacked will power to actually program it. (I couldn’t believe these guys!!)

Data population is so common a concept that many Back end APIs provide ways to get only selected properties. For example, there could be a better endpoint like this:

getProfiles(page, { 
    populate: ['documents', { 
        populate: ['title'] 
    }]
});