Add proxy code by rtsisyk · Pull Request #1 · organicmaps/url-privacy-proxy

rtsisyk · 2023-07-12T08:54:04Z

No description provided.

rtsisyk

The general recommendation here: please try to refactor the code to isolate different cases and comments about each case. Currently, it is really hard to understand what is parsed, when and how.

rtsisyk · 2023-07-12T08:58:18Z

+      - run: npm run format
+      - run: git status
+
+


nit: fix trailing space and keep only one empty line

rtsisyk · 2023-07-12T08:58:33Z

@@ -0,0 +1,2 @@
+node_modules/
+node_modules


node_modules/ should be enough

add empty line

rtsisyk · 2023-07-12T09:00:13Z

+    decodedURI = decodeURIComponentTillSame(decodedURI);
+    return decodedURI;
+  }
+  function decodeURIComponentTillSame(uri) {


What is the idea behind running decodeURI() in a loop?

Sometimes JavaScript decode utility doesn't decode the url in one go. For some URIs, the request throws highly encoded URIs which have to be decoded further to make a further request or extract information from them.

Could you please provide a link to documentation that describes this limitation of decodeURI()? Yes, I understand that a URL can be encoded twice or even thrice, but is it a case?

rtsisyk · 2023-07-12T09:02:51Z

+      DNT: '1',
+    },
+  })
+    .then((value1) => {


What does this block do?

It resolves the async promise.

A silly question: why do we have both ".then((value1) => {" and " .then(async (value2) => {"?

Sorry I am not following why both these then blocks are needed. Can we say const resp = await fetch(request.query.url, { and then use resp.url ?

rtsisyk · 2023-07-12T09:13:52Z

+          lati = final.plus_code.geometry.location.lat;
+          lng = final.plus_code.geometry.location.lng;
+        } catch {
+          if (host === 'www.google.com' || host === 'maps.google.com') {


what case is handled here?

As in the code below this also the case handled here is when the coordinates are stored in a link which contains ";markers" in it which are located in the backend.

I am surprised that we check the host only here. Shouldn't it be checked at the beginning of the function? This link extended, maybe extended for other providers in the future.

rtsisyk · 2023-07-12T09:15:56Z

+          console.log(global_code);
+          var global_code = result.plus_code.global_code;
+          console.log(global_code);
+          var pc_final = global_code.substring(0, 4) + plusCode.substring(0, plusCode.length);


Magic offsets. Please provide an example of strings parsed in a comment.

rtsisyk · 2023-07-12T09:18:24Z

+                  lati = lati.toString()
+                  lng = lng.toString()
+                } catch {
+                  console.log('third')


Too many nested levels... could you please describe possible cases here in bare English to understand how to streamline this code?

Maybe we can create different functions and in place of try catch we can call these functions?

rtsisyk · 2023-07-12T09:18:47Z

+                  console.log('fifth')
+                  let searchString = 'https://www.google.com/maps/search/'
+                  position = searchString.length + results.indexOf('https://www.google.com/maps/search/')
+                  link = results.substring(position,position+250)


Again, magic offsets.

I will add more comments for better readability and understanding

rtsisyk · 2023-07-12T09:19:17Z

+                try {
+                  console.log('fourth')
+                  position =results.indexOf('https://www.google.com/maps/place/')
+                  link = results.substring(position - 1, position + 250);


magic offsets

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

rtsisyk · 2023-07-21T09:12:24Z

+  var lng = null;
+  var urllen = request.query.url.length;
+  var string = '';
+  for (const key in request.query) {


Could you please explain the logic behing this code?

rtsisyk · 2023-07-21T09:13:40Z

+      DNT: '1',
+    },
+  })
+    .then((value1) => {


A silly question: why do we have both ".then((value1) => {" and " .then(async (value2) => {"?

rtsisyk · 2023-07-21T09:17:23Z

+      const array = url.split('/');   // Stores an array which contains parts of the url which are separated if they have an "/" between them. Example:- ""https://maps.google.com?q=VGR4+5MC+Falafel+M.+Sahyoun,+Beirut,+Lebanon&ftid=0x151f16e2123697fd:0x8e6626b678863990&hl=en-US&gl=tr&entry=gps&lucs=47067413&g_ep=CAISBjYuNjQuMxgAINeCAyoINDcwNjc0MTNCAlJV" is split into array whose array[arrLength-1] will be "maps.google.com?q=VGR4+5MC+Falafel+M.+Sahyoun,+Beirut,+Lebanon&ftid=0x151f16e2123697fd:0x8e6626b678863990&hl=en-US&gl=tr&entry=gps&lucs=47067413&g_ep=CAISBjYuNjQuMxgAINeCAyoINDcwNjc0MTNCAlJV" (in some URIs this part is even smaller) as you can see this part contains plus code and address in this specific use case which will be utilised further.
+      const arrLength = array.length;
+      try {
+        // PLUS CODE MECHANISM UTILISED:- In the following lines of code firstly the address is separated from plus code. The plus code we get is a shortened plus code so it won't be of use untill we expand it. For this we need to understand plus code making mechanism in itself.


How URLs with pluscodes are detected? The code above assumes that q= parameters is a plus code. Wasn't it better to check that q= is a plus code before starting to parse it?

rtsisyk · 2023-07-21T09:17:57Z

+          return response.json();
+        }
+        const response = await fetch(`https://geocode.maps.co/search?q=${cityandState}`); // This send an API request to retreive the center of the city.
+        const results = await gatherResponse(response);


why await response.json() is not enough?

rtsisyk · 2023-07-21T09:19:47Z

+            return response.text(); // Returns the HTML code.
+          }
+          var response = await fetch(url); 
+          var results = await gatherResponse(response); // results contains the HTML code.


ditto: await response.text().

rtsisyk · 2023-07-21T09:22:24Z

+// The logic of the proxy is that in the back HTML code of the site, there may lie some different URIs which contain the coordinates, although it is hard to predict which one will be there so we check one by one by using try catch blocks. Also extracting coordinates from plus codes is also utilised which may not require the back HTML code.
+const cheerio = require('cheerio');
+
+export async function getCoordinates(request) {


What do you think about splitting this function to separate routines for each use-case? The current version is too big and hard to read.

rtsisyk · 2023-07-21T09:24:20Z

+          lati = final.plus_code.geometry.location.lat;
+          lng = final.plus_code.geometry.location.lng;
+        } catch {
+          if (host === 'www.google.com' || host === 'maps.google.com') {


I am surprised that we check the host only here. Shouldn't it be checked at the beginning of the function? This link extended, maybe extended for other providers in the future.

rtsisyk · 2023-07-21T09:25:00Z

+  lati = decodeURIComponent(decodeURIComponent(lati.toString())).trim();
+  lng = decodeURIComponent(decodeURIComponent(lng.toString())).trim();
+  console.log(lati, lng);
+  if (lati.charAt(0) === '+') {  // Sometimes coordinates are extracted as +24.678,89.909 or +23.546,-12.845. And in these type of coordinates a positive or negative sign accompanies them, while the negative sign seems to fit with the geo URI scheme (because of negative coordinates), the positive sign isn't so we have to remove the positive sign.


normalization of coordinates can be extracted into a separate function

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

rtsisyk · 2023-08-15T09:09:40Z

+  reqBody = null;
+  lati = null;
+  lng = null;
+  urllen = request.query.url.length;


This line fails if no url= passed. You can return 400 Bad Request here for this case.

http://127.0.0.1:8787/coordinates

rtsisyk · 2023-08-15T09:11:08Z

+  return decodedURI;
+}
+
+async function gatherResponse(response) {


This function is defined twice.

rtsisyk · 2023-08-15T09:22:58Z

@@ -0,0 +1,24 @@
+const axios = require('axios');


axios is not in "devDependencies" in package.json

rtsisyk · 2023-08-15T09:24:13Z

+let position;
+let results;
+
+async function gatherResponse(response) {


This wrapping function is not really useful. Please just replace async gatherResponse(response) with async response.json() and the same for gatherResponseText().

rtsisyk · 2023-08-15T09:27:06Z

@@ -0,0 +1,371 @@
+const cheerio = require('cheerio');
+
+let urlObj;


Using global variables is considered a bad practice, in any programming language. Please define variables where they are needed (by using let). Use function arguments to pass all values that are needed for the work to do its job. Avoid global state.

rtsisyk · 2023-08-15T09:54:29Z

+    console.log(arrLength);
+    const qParam = urlObj.searchParams.get('q'); // This gets the qParam from the above mentioned part of URL and then splits them apart into plus code and address
+    if (qParam) {
+      console.log(array[arrLength - 1].split('=')[1]);


Any examples please.

Example:- https://maps.google.com/maps?q=9CRJ+XJC+Division+Of+Physiology+%26+Climatology,+IVRI+Mini+Bye+Pass+Road,+Bareilly,+Uttar+Pradesh&ftid=0x39a006fb82608637:0x944d17bf7a6130e9&hl=en-IN&gl=in&entry=gps&lucs=,47059802&g_ep=CAISBjYuNjYuMxgAIKzfASoJLDQ3MDU5ODAyQgJJTg%3D%3D&g_st=ic

rtsisyk · 2023-08-15T09:54:57Z

+    }
+    link = decodeURITillSame(address);
+    console.log('here2');
+    var cityandState = link.split(',')[link.split(',').length - 2] + link.split(',')[link.split(',').length - 1]; // Now this part stores the city and state/province of which the address is of.


Ditto, please provide any example of a string that is being parsed here.

rtsisyk · 2023-08-15T09:55:33Z

+    console.log(response);
+    var lat = results[0].lat;
+    var lon = results[0].lon;
+    const res = await fetch(`https://plus.codes/api?address=${lat},${lon}&email=kartikaysaxena12@gmail.com`); // Utilises the latitude and longitude to retreive the plus codes of the center of the city.


Is email required? Can we remove it? If not, please create a global const.

Yes that's given in the documentation. I would create a global constant.

rtsisyk · 2023-08-15T09:56:04Z

+    var lat = results[0].lat;
+    var lon = results[0].lon;
+    const res = await fetch(`https://plus.codes/api?address=${lat},${lon}&email=kartikaysaxena12@gmail.com`); // Utilises the latitude and longitude to retreive the plus codes of the center of the city.
+    const result = await gatherResponse(res);


here and in all other places:

Suggested change

const result = await gatherResponse(res);

const result = await res.json();

rtsisyk · 2023-08-15T09:56:43Z

+    const res = await fetch(`https://plus.codes/api?address=${lat},${lon}&email=kartikaysaxena12@gmail.com`); // Utilises the latitude and longitude to retreive the plus codes of the center of the city.
+    const result = await gatherResponse(res);
+    var global_code = result.plus_code.global_code;
+    var pc = link.split('+')[0] + '%2B' + link.split('+')[1];


What is going on here? Please elaborate.

Generally the plus codes retreived from the URIs are short codes, example JJXX+HR8, Seattle contrary to the long plus codes example 7MQ6GVR2+99. These contain a part of code and the address nearby. The JJXX+HR8 denotes the accuracy of the coordinates upto last decimal in address namely Seatltle. So I use Geocoding APIs to retrieve the centre of the address provided along with the plus code, which is Seattle over here and change the last part of code to prepare the plus code of our location.

For example "Q5RX+VX6, New Colony, Ayodhya, Uttar Pradesh 224123" is the plus code returned by the URI, Now I utilise the address via Geocoding API to retrieve the centre plus code of "New Colony, Ayodhya, Uttar Pradesh 224123" which returns "7MR4Q6V3+QV". Now I use "7MR4Q6V3+QV" and "Q5RX+VX6" (which was earlier received ) to prepare the plus code which is accurate which is "7MR4Q5RX+VX6".

Now this step is a little bit lengthy and can be substituted via Google API but for unshortening short plus codes it has a paid API. I also looked up npm packages but didn't find one that suits our requirements.

My understanding that var pc = link.split('+')[0] + '%2B' + link.split('+')[1]; basically takes 7MR4Q6V3+QV and replaces it with 7MR4Q6V3%2BQV (an url encoded version).

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

Add proxy code

872fe22

rtsisyk commented Jul 12, 2023

View reviewed changes

rtsisyk requested a review from kartikaysaxena July 12, 2023 09:27

rtsisyk assigned kartikaysaxena Jul 12, 2023

kartikaysaxena added a commit that referenced this pull request Jul 12, 2023

#1 Added comments and minor changes

47f9835

kartikaysaxena added a commit that referenced this pull request Jul 13, 2023

(#1) Added some more comments

a2e2ad4

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

Updated code

d128006

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

rtsisyk commented Jul 21, 2023

View reviewed changes

Refactored by adding functions

dadf09f

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

rtsisyk commented Aug 15, 2023

View reviewed changes

kartikaysaxena added 2 commits August 27, 2023 18:39

Suggestions implemented

4adf393

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

Some more minor changes

f51396e

Signed-off-by: Kartikay <kartikay_2101ce32@iitp.ac.in>

		@@ -0,0 +1,2 @@
		node_modules/
		node_modules No newline at end of file

		@@ -0,0 +1,371 @@
		const cheerio = require('cheerio');

		let urlObj;

	const result = await gatherResponse(res);
	const result = await res.json();

Conversation

rtsisyk commented Jul 12, 2023

Uh oh!

rtsisyk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment