Opened PR for three of the locker providers: Omniva, Venipak and Smartpost/Itella.
Those not added:
\* : pun very much intended
"Unisend is prohibited on my laptop"?
Ah, yeah shared a bit more details in ticket. But here it is:
As I understand Unisend is an umbrella for parcel lockers branded as Unisend and Unisend uDrop.
Site in the description only has later ones.As far as I can tell, unisend.lv has its own map with both lockers there. But my laptop prevents me from accessing that site: it considered a security threat (don't have details as to why).
Whenever I try to open unisend.lv, one of the security tools prevents me from doing it. This limitation is simply here, and cannot do anything about it at the moment.
Probably if somebody would be kind enough to identify request to get locations, and will send me data file with it, I probably can place it into cache manually.
Does https://my.unisend.lv/map work for you or same issue?
Data appears to be at https://api-esavitarna.post.lt/terminal/list?size=9999
list.json
Here is the JSON content/response
markalex2209 said:
- DPD has only addresses
Yeah, this sucks. But it theoretically possible to parse addresses as Latvia has all official addresses imported. The biggest problem will be all the broken syntax they will likely have. I have a partial hard-coded address string to actual OSM address matcher already.
markalex2209 said:
- Considering what will be the correct approach here: would prefer to parse it all in the same place.
Parsing it in the same place is totally the way to go, yeah. Analyzers can just get whatever specific data they need..
markalex2209 said:
- And then either use in three different analyzers, or put all of that into a single report.
I think it would be better to put at least parcel lockers in separate analyzers so that all parcel lockers share a common base class like you are already doing - then we can present them all similarly in the new parcel lockers section. I feel like post boxes and post offices are also fairly distinct and can be put in separate reports - they have a bunch of tagging stuff we can theoretically check. Since they are probably all using generic correlation logic, they are just copy-paste boilerplate code mostly anyway
Same issue for my.unisend.lv
. But api is accessible, thanks for the link.
Will try to add analyzer for it shortly.
HellMap said:
markalex2209 said:
- DPD has only addresses
Yeah, this sucks. But it theoretically possible to parse addresses as Latvia has all official addresses imported. The biggest problem will be all the broken syntax they will likely have. I have a partial hard-coded address string to actual OSM address matcher already.
I can't say that I have full understanding of what is present now.
My current understanding of how this is possible is:
If this is correct - it feels like to much hustle for now, TBH. There are a lot of low hanging fruits=)
Oh yeah, for sure. Stuff like this gets complicated quickly. There's a bunch of online maps we can make quick correlations with. I might check out DPD later and see if it's viable.
We do only have 5 (now 6 I guess with Unisend) major parcel locker companies though, so it would be.... interesting omitting what is essentially the biggest one in world scope sense
HellMap said:
We do only have 5 major parcel locker companies though, so it would be.... interesting omitting what is essentially the biggest one in world scope sense
Well, if they don't want to have map of their lockers, as mush as not to provide it on the site, who are we to force appearance of said lockers on any other map?)
On one hand, yes. But, gee, you should see what I do for some data - literally running chrome browser and clicking on navigation buttons to scrape their live request data because they block everything and anything trying to interact directly. I think it was one of the banks.
HellMap said:
On one hand, yes. But, gee, you should see what I do for some data - literally running chrome browser and clicking on navigation buttons to scrape their live request data because they block everything and anything trying to interact directly. I think it was one of the banks.
Yes, I've seen selenium being in dependencies, and guessed something like that was a reason.
By the way, DPD has some form of map, where you can put in your address/index and it'll show lockers nearby. But I don't see any reasonable way to use it.
@HellMap, Osmalyzer skipping Venipak for some reason. Can you check why?
When I run it locally, everything works fine.
image.png
So the default web read fails on GitHub with
Failed to retrieve with exception!
Exception message: One or more errors occurred. (The SSL connection could not be established, see inner exception.)
Inner exception message: The SSL connection could not be established, see inner exception.
Inner exception message: Authentication failed, see inner exception.
Inner exception message: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
Inner exception message: error:0A000410:SSL routines::sslv3 alert handshake failure
I switched it to browsing instead of raw read as that sometimes helps, but whatever reply it got wasn't formatted as expected - it probably got some error page or something like Cloudflare protection or something.
Exception message: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
Probably means it was a start of some sort of html instead of JSON
Hmm, okay, so browsing response wraps it up into
We were trying to parse: <html><head><meta name="color-scheme" content="light dark"><meta charset="utf-8"></head><body><pre>[{"id":3348,"name":"Venipak locker, Coop Venipak pakiautomaat","code":"12142751","address":"P\u00e4rn [1635718]...
So it sounds like the data is there, just... decorated. This doesn't happen on my local machine.
Okay, I stripped the html from the response and it seems to work fine now. I have no idea why Github gets it wrapped in HTML and locally I don't. Or why their SSL fails with this site by default. I imagine whatever fails by default is what full Chromium/Selenium browser knows how to handle, but the result is different. I am not even going to try and debug this.
To resume from https://github.com/OSMLatvija/Osmalyzer/pull/33#discussion_r1619420968
One thing that needs an alignment: how both of them should be tagged?
I assumedbrand=Unisend
+operator=Unisend
andbrand=Unisend
+operator=uDrop
(and already mapped a couple in that way). But maybe this is a correct usecase fornetwork
=Unisend
?
This is something I have never encountered. In fact, I wonder if anyone has? Like, are there any other companies in the world that offer parcel locker "rental" like this, so to speak. I have no idea how we are going to tag this. Like, what happens if a parcel locker is used by more than one service? We currently only have Unisend -branded lockers from uDrop, so it's almost fine to do brand=Unisend
+ operator=uDrop
. But then what happens with brand=Unisend;Superpackages;BoxShippers
+ operator=uDrop
?
HellMap said:
Okay, I stripped the html from the response and it seems to work fine now. I have no idea why Github gets it wrapped in HTML and locally I don't. Or why their SSL fails with this site by default. I imagine whatever fails by default is what full Chromium/Selenium browser knows how to handle, but the result is different. I am not even going to try and debug this.
It could be the Content-Type header or similar differences in how requests are sent. One could play with curl and see whether it explains the HTML vs JSON diff (and then possibly set that header in Github, if supported).
SSL - could be missing the CA cert, or perhaps some validation differences. Can be debugged, but if there's a workaround... :)
HellMap said:
This is something I have never encountered. In fact, I wonder if anyone has? Like, are there any other companies in the world that offer parcel locker "rental" like this, so to speak. I have no idea how we are going to tag this. Like, what happens if a parcel locker is used by more than one service? We currently only have Unisend -branded lockers from uDrop, so it's almost fine to do
brand=Unisend
+operator=uDrop
. But then what happens withbrand=Unisend;Superpackages;BoxShippers
+operator=uDrop
?
I'd say make it work for the current situation, worry about potential scenarios once they appear.
How would you debug something that is running on a Github machine? For example, I can't "play with curl", because that means making a repo, setting up actions, writing that custom curl code and runners to actually run it and it's not like I can debug in memory - all I get is output. And who says curl will work the same on Github as it works locally? So it's absolutely not worth the effort.
I meant using curl elsewhere to figure out whether HTTP headers can predictably affect output.
(or any other tool, just that curl would be my first choice)
HellMap said:
So the default web read fails on GitHub with
Failed to retrieve with exception! Exception message: One or more errors occurred. (The SSL connection could not be established, see inner exception.) Inner exception message: The SSL connection could not be established, see inner exception. Inner exception message: Authentication failed, see inner exception. Inner exception message: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. Inner exception message: error:0A000410:SSL routines::sslv3 alert handshake failure
I believe usual two suspects will be incorrect time(if github runner does something strange with their machine's time) and difference in supported protocols (if github enforses newer versions of tls then the one used on venipak's site).
I don't think we can do much in either case, but I'll try to dig around a bit just out of curiosity.
Regarding selenium - I'd guess this wrapping of json into html is something what browser could do. Don't have recent experience with selenium, but might be that your local setup uses different engine/version/options than one at github.
For uDrop: I agree with Rihards. Let's decide based on how it is now, and we'll adjust if needed later.
If we'll enforce reasonable tagging, it should be rather easy to batch retag into a different way later. And I use word enforce loosely here.
markalex2209 said:
Regarding selenium - I'd guess this wrapping of json into html is something what browser could do. Don't have recent experience with selenium, but might be that your local setup uses different engine/version/options than one at github.
The thing is, it doesn't wrap it in html on local machine - it returns raw json as expected. May be there's some difference between Selenium versions or OS-specific stuff or just somewthing github does. But regardless, it's unlikely something we can do anything about.
markalex2209 said:
it should be rather easy to batch retag into a different way later
Famous last words before tagging scheme gets stuck forever. But, yeah, we would probably do the "normal" stuff for now. I am just thinking about potential future problems.
HellMap said:
The thing is, it doesn't wrap it in html on local machine - it returns raw json as expected. May be there's some difference between Selenium versions or OS-specific stuff or just somewthing github does. But regardless, it's unlikely something we can do anything about.
Can Github stuff specify HTTP headers?
Github doesn't specify anything, it just runs the code. We specify headers in code. Or technically, it's all defaults currently. The same code runs on Github, just in a different environment. You aren't even supposed to specify headers, that's the point of headless browsing - to receive what the user would receive. More to the point - what headers would you want to specify/change?
Oh, it's running a browser behind the scenes? Yeah, then it's weird, I thought it's some code/CLI solution directly requesting data.
Yeah, as mentioned above from those errors, "direct" connection simply fails (as much as http client is a "direct" connection), so have to use a "full" browser here
@HellMap, I need a clarification what exactly you meant by
Latvijas Pasts (also has mail boxes)
in this issue?
Because I initially that that you refer to the usual mail boxes being listed on the same map with parcel lockers.
But yesterday I noticed that a lot of LP parcel locker seem to be equipped with built-in mail boxes:
20240530_223150.jpg
And now I'm a bit in doubt if I understood you correctly.
Well, I was not aware that exists
https://pasts.lv/lv/zinas/7875-latvijas-pasts-atklaj-pirmos-pakomatus-ar-vestulu-kastitem This seems very new
Ok, I'm glad that at least my initial understanding was aligned with yours)
But now we need to figure out how to tag this stuff(
I think this is world-first.... at least I cannot find anyone else already doing it.
I guess we'll have to invent something, probably letter_mail_in=yes to match parcel_mail_in=yes
Since we cannot tag mail box on parcel locker, as both are amenity and then every consumer would break with 2 values
According to https://www.parcelandpostaltechnologyinternational.com/analysis/analysis-sending-letters-via-parcel-locker-will-it-catch-on.html InPost allows sending letters between parcel lockers, not entierly sure what that means
Video seems to indicate that InPost one and "ours" are different. I'd even say that InPost one doesn't require additional tagging, since they still operate in the same way as with packages, it's just that that "package" is very thin.
But for LP this seems to be indeed an additional mail box, and then letters are handled as usual.
I like your idea with letter_mail_in=yes
. Seems very reasonable and easy to adopt.
Might be consumer-unfriendly, especially if this is unique to Latvia.
Maybe a two node solution?
Not great, but maybe as a stop-gap...
We shouldn't really be setting precedent for breaking the core principle of one feature, one element. And this really should be an OSM proposal so everyoen is aware.
Well, remotely similar is a bank branch with an ATM, where a widely accepted (and best even) practice is to map them separately - we could say that parcel locker is a bigger feature like bank branch, that has a smaller feature like ATM - letterbox ;)
Not something I'd push 100%, but there seems to be some similarity.
Rihards Olups said:
Well, remotely similar is a bank branch with an ATM, where a widely accepted (and best even) practice is to map them separately - we could say that parcel locker is a bigger feature like bank branch, that has a smaller feature like ATM - letterbox ;)
Not something I'd push 100%, but there seems to be some similarity.
I disagree with this analogy. ATM is mapped separately to differentiate between tags of ATM and tags of branch. And additionally, to specify locations, if branch size too large.
Here the object is inseparable.
To add, by that logic, we should be placing a mailbox and a parcel locker on every post office.
HellMap said:
To add, by that logic, we should be placing a mailbox and a parcel locker on every post office.
Aren't we? I mean mail boxes are usually outside and act differently from post office.
But let me maybe try a different approach to this problem. If parcel locker and mailbox were standing at the same place - would we tag them as two?
And if a usual mail box was hanged not on the wall, as it usually is, but on the side of parcel locker - would we then tag it as two?
markalex2209 said:
HellMap said:
To add, by that logic, we should be placing a mailbox and a parcel locker on every post office.
Aren't we? I mean mail boxes are usually outside and act differently from post office.
Yes, but I mean, another mail box node where the counter is - because you can mail in directly with the person. And you can receive packages over the counter. These are obviously not the same objects as the separate mailbox and any PO boxes they have.
I suppose if they did literally hang the mailbox on the side of the parcel locker and they didn't interoperate in any way, I would personally tag it as two nodes
(By the way, I know we inter-replied to each other, so I might have made things more confusing. I was also replying to rich with a contrived example.)
HellMap said:
I suppose if they did literally hang the mailbox on the side of the parcel locker and they didn't interoperate in any way, I would personally tag it as two nodes
Well, I don't believe those two interoperate in any way. It's simply an additional locked space with a slit in a door.
I don't see why we would want to map it differently, just because mailbox is slickly hidden under the screen, and not nailed to the sidewall.
I initially didn't thought this would result in a tagging scheme discussion. Shouldn't this part of chat be extracted and moved to #general
?
Probably... this should be a proposal on OSM technically if we are to introduce a new tag or consider double-tagging.
23 messages were moved from this topic to #general > Latvijas Pasts pakomātu vēstuļu sūtīšana / parcel locker ... by HellMap.
Urgh, after renaming the target topic (because the title got truncated due to length), it's no longer pointing to the right link. But it's #general > Latvijas Pasts pakomātu vēstuļu sūtīšana
Added PR for Latvija Pasts lockers and mail boxes.
Scraped data also contains all other items from LP site. I believe it can be used directly for a couple tasks in #33, but I don't fully understand what is the vision there, so will not try anything with it.
Cool, I 'll check it out. We probably want mailboxes separate because they have stuff like ref codes and collection times and we might want to check their tags and may be do some stats or something.
I think Circle K is just parcel receival because LP doesn't have officies nearby
Any additional fields should be fairly easy to add if needed.
HellMap said:
I think Circle K is just parcel receival because LP doesn't have officies nearby
I guess so.
But at least in Riga a lot of them also have lockers, so not sure.
I've made also unisend, but found that their data also contains provider cleveron
. No idea what to do with this fact at this point.
tableConvert.com_j71g7t.xlsx
I think I've seen one of them, and don't recall any additional markings, but I'm basically not sure in anything for this network RN.
Unisend-Parcel-lockers-report.html
Cleveron seem to be parcel locker manufacturer https://cleveron.com/products/cleveron-352
But yeah, I also do not understand what they mean with their "provider" mixing "cleveron", "udrop" and "unisend"....
HellMap said:
So the default web read fails on GitHub with
Failed to retrieve with exception! Exception message: One or more errors occurred. (The SSL connection could not be established, see inner exception.) Inner exception message: The SSL connection could not be established, see inner exception. Inner exception message: Authentication failed, see inner exception. Inner exception message: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL. Inner exception message: error:0A000410:SSL routines::sslv3 alert handshake failure
Checked TLS protocols supported by venipak: nmap --script ssl-enum-ciphers -p 443 go.venipak.lt
Starting Nmap 7.95 ( https://nmap.org ) at 2024-06-02 17:39 FLE Daylight Time
Nmap scan report for go.venipak.lt (46.255.211.5)
Host is up (0.0098s latency).
PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
| TLSv1.0:
| ciphers:
| TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
| compressors:
| NULL
| cipher preference: server
| warnings:
| Broken cipher RC4 is deprecated by RFC 7465
| Forward Secrecy not supported by any cipher
| TLSv1.1:
| ciphers:
| TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
| compressors:
| NULL
| cipher preference: server
| warnings:
| Broken cipher RC4 is deprecated by RFC 7465
| Forward Secrecy not supported by any cipher
| TLSv1.2:
| ciphers:
| TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
| TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
| TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_256_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
| TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
| TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
| TLS_RSA_WITH_CAMELLIA_128_CBC_SHA (rsa 2048) - A
| compressors:
| NULL
| cipher preference: server
| warnings:
| Broken cipher RC4 is deprecated by RFC 7465
| Forward Secrecy not supported by any cipher
|_ least strength: C
Nmap done: 1 IP address (1 host up) scanned in 2.40 seconds
While those are OK, it's a bit sus, that only RSA is in the list.
Quick check through other sites used showed that other sites have other TLS ciphers supported.
Have to guess that those RSA ciphers are disabled on build agent for some reason. Handshake error is inline with this.
Only thing I don't know is why browsers works. Most of browsers (only exception I know of FF and its "family") use OS protocols.
If my conclusion is correct, there is not much of what we can do from our side: doubt you can control settings of OS.
There might be some stuff one could control, but I am personally really not going to bother with this. This is just too much work and it's all very fragile. Sites break and change all the time and if they don't work with default setup, then theres only so much effort I'm willing to spend. I only decided to investigate RS site because I wanted to know if this is something I can just bulk-fix once for the future if this is really going to be a common problem.
There's https://stackoverflow.com/a/71672975/272387 , but meh...
As far as I can tell, GITHUB_ACTIONS_RUNNER_TLS_NO_VERIFY=1
is for a different purpose: to skip cert verification for connection between build agent and Github.
Maybe there is something similar for general Http connections, but I don't think it would help us either.
At this point problem is that client cannot finish handshake (agree with server on protocol+cipher to establish a connection). This happens before certificate validation, IIRC. I don't know .net enough to state anything authoritatively, but I doubt that it's possible to override list of supported algorithms from within app (at least for default http library).
I guess(!) it's possible to use some different http lib, that doesn't rely on OS implementation/configs.
Ah, indeed. I guess this stays open for now then.
This is a real DHL locker: have seen it with my own eyes. Should DHL be added into the list of "expected brands"?
Wondered how many are there in Latvia, but the map in https://dhlexpresslatvia.lv/lv/dhl-paku-terminalis/ does not load for me (the "allow" button does nothing).
Same
Called DHL, it's their only parcel locker / mail-in station in Latvia.
A site where the map works: https://mydhl.express.dhl/lv/lv/locator.html .
Although cannot find that locker in the map.
Yeah, should add to the list. I know we also have a couple lockers that do not belong to any of the big networks but are more like "you can pick ups stuff from this shop after hours".
For example, https://www.mapillary.com/app/?focus=photo&pKey=611575394153040 that I finally remembered to map https://www.openstreetmap.org/node/11953641656
Yes, I had noticed something like this in one of the pharmacies. Not sure though if it's correct to call one of those a parcel locker.
I'm pretty sure it is - I mean, it's a storage locker for... parcels :) We don't have many that aren't from the main brands, let alone have them mapped. But I don't see why they wouldn't count - and what other tag would we choose?
8 messages were moved from this topic to #general > Pakomātu dažādība by HellMap.
hi! you could get dpd parcel locations from json that is available here: https://www.dpd.com/ee/en/business-customer-solutions/
the "Automatically updated parcel locker list" section
Nice find!
Yes, that is exactly what is needed, will add those.
Unless you want to implement it yourself)
Neat, thanks. The opening hours format/contents is a bit weird, but that's probably not the first priority.
DPD opening hours
We are ignoring opening hours for all of the lockers at the moment.
Looks like all items are the same for opening times - 24/7. This seems more like someone trying to work around the scheme they had to use. But yeah, not using these yet. I think the ones that might have opening hours are inside shopping centers and such, so their hours match those. it would probably be an error to tag them as anything other than those hours.
Parsed them https://osmlatvija.github.io/Osmalyzer/DPD%20Parcel%20lockers%20report.html Thanks for finding the data file
Yeah, absolutely.
Are you saying that some DPD lockers are inside shopping centres, but their data does not reflect that?
Not specifically DPD, just lockers in general. I think it's only those stupid Smartpost ones that are always inside. Unfortunately, it's what Amazon uses, so it's a pain...
Also quite a lot of Latvijas Pasts ones are inside stores.
Oh, I hadn't seen them. I guess I just don't visit stores :D
I didn't know until I needed to receive something from one, and struggled to find it)
Though my "usual" one is technically inside, between two entrance doors.
Last updated: Dec 22 2024 at 05:10 UTC