Proxy in front of photoprism that limits response size?
but it should not limit response time
hey all! I know the devs are on holiday, but has anyone else seen this issue... yesterday I scanned 43 old photos and added them to photoprism. At first they all showed up without issue, locally and remotely via shared link. I sent the share to two people and the first reported everything was fine while the second said 'some' were missing. Today when I checked I only see 6 photos remaining in the album. All the sidecar files are there and the original .tif files all show up in the Originals folder, it just seems like that folder 'lost' 36 of my photos. And now that I check again, they're all gone and the folder is missing in 'folders' (again, .tifs are still present in the Originals section of Library). Re-running a scan does nothing, re-running a Complete Rescan suggests PP knows the thumbnails are there (it intentionally doesn't create new ones) but there are a lot of errors saying no jpg for an underlying file. When the Complete Rescan finishes the photos pop back up, but I have NO idea where they went the first time and what to do to keep them from disappearing in the future...
No idea without getting remote access when it happens or instructions how to replicate this locally for debugging. Might be many reasons incl hardware and file system issues.
That said, never seen something like this happening and never got similar reports.
I do wonder whether it's in some way related to Unraid and how I'm writing things to a cache drive and then having them offload onto HDDs. The files are symlinked somehow so that shouldn't be an issue, but I'll see if I can keep an eye on the logs and see if anything obvious pops up
2 replies
Could those be duplicates? Did somebody move them to archive? Did they get stacked? Do you have the quality filter enabled (check the resolution of missing files)?
Just came back from an 8h drive :) happy to check specific issues if you can provide us with details incl a complete description of your environment, like the symlinks, caches and drives you use
It may happen when the indexer runs while external file system operations are running
That's why we don't index automatically by default
Folder could be in an inconsistent state for a short time
I'll try to gather more info-if it's not something you've heard from before it's possible it's just an unraid problem, but what was curious for me was it worked, then I sent out share links and went to bed, and some family members reported nothing amiss but others said they couldn't see most of the files. When I woke up I saw only 6/42 files remained, after I cycled through those remaining 6 the folder itself disappeared
without any obvious errors in PP
No, typically it's stacking, file permissions or the quality filter when we receive similar reports
Symlinks, especially dynamic ones, sometimes also cause issues as mentioned above. For example when they point to folders or files not visible inside the Docker container.
That's not a photoprism specific issue though.
let me try to gather more info-was just seeing if anyone else had seen similar issues. I have another batch of photos I want to scan and send out-let me see if that has the same issue. I have a fair few charlie-brown-icon broken files in my setup but I haven't seen images flat-out disappear like this before, so it just struck me as odd. To confirm, though, a complete re-index of that sub-folder did seem to fix it, so until I'm certain it's reproducible I don't want to waste your time
hope you had a good weekend away!
There are real solutions to caching. Linux already buffers read files in RAM, you can increase this a lot. Moving SWAP to SSDs You can increase SWAP likely hood a lot as well. Also dm-cache which is less known https://www.redhat.com/en/blog/improving-read-performance-dm-cache and https://en.wikipedia.org/wiki/Dm-cache (if it isn't in the kernel or a kernel module it's either slow or using some sloppy method, in this case, it's sloppy, not really "cache" at all)
Yes but it is a bad implementation of what already exists in the kernel. They should have used dm-cache under the hood. I've seen it break basically every software I've even been slightly involved in at one point.. (and seems like you might be able to add this one to that list? lol)
If it is another software with the issue because of that, I think it's time to have a strong talk to them since so many are spending money on unraid lol..
Maybe at the beginning it breaks the files and then they return or something, idk, not gonna try unraid or anything. Files disappearing is definitely no indexer's friend
This is probably the simplest tutorial, lvmcache uses dm-cache under the hood. https://forum.level1techs.com/t/lvmcache-a-simple-how-to/133247
I'm moving everything to the array (HDD) for now to see if that helps solve the issue. Near as I can tell there are files spread between the cache drive and array, so perhaps it's just an unfortunate confluence between files being half-moved from cache to array. My ideal situation would be to have PP's sidecar/thumbnails on the SSD 'cache' drive, but just permanently living there for the increased speed of the cache, it's just that comes out to almost 100GB for me so I need to get an increased cache drive first...
Yea that sounds like a good setup. On first view of a full size photo you might hit drive spin up time though so it could request timeout since that takes like 15-20 seconds, keep in mind. (not sure what your http timeout is set to)
yeah, that's a valid point-the intent is to move to a SSD drive regardless, but that's a down-the-line upgrade so I'll see how the HDD works for now. At the moment I'm wiping out all the sidecar data, running 'photoprism reset' and starting a complete re-index with that unraid share set manually to a single array drive to avoid spreading files out across multiple ones to see if that helps...
it does appear to work just fine. I'm going to upgrade to a 1TB cache drive and once I've done that move PP over to a cache-only share so it'll still live on a single drive but a cache'd one so hopefully it'll be faster and less draining on the HDD, because PP seems to do a lot of housekeeping
Hi, I'm trying to set up photoprism on my rapsberry pi but am stuck. I can't seem to install Docker Compose (using this guide: https://linuxhint.com/install_docker_raspberry_pi-2/), as it gets stuck here: "Running setup.py bdist_wheel for pynacl ... " I tried to fix this using the steps outlined here (https://www.reddit.com/r/docker/comments/bpxjtp/trying_to_install_dockercompose_on_rpi_3_b/enyncvg), but that didn't help. :(
I was able to fix this by just letting it run for about 40 minutes. It's weird that it took so long, but I'm glad that I can continue now :)
What should I set " PHOTOPRISM_SITE_URL" to? I tried the default value ("http://localhost:2342/"), but entering "http://localhost:2342/" and "[IP of my RPI]:2342" in the URL bar of my browser didn't work. I also tried to set it to the IP of my RPI and made sure to reserve its IP through my router settings, but that didn't work either. Sorry for all these messages, as you may be able to tell I'm new to all this.. 😅
I used the container inspect command ("sudo docker container inspect [ID of relevant container]") and found out that, apparently, its IP is outside my home network...? i.e. all devices in my home network have an IP that starts with 192.168.1.x, but according to this, the IP is completely different. 🤔
It's weird that it took so long
You're using a raspberry pi 2...
I tried localhost
From the pi 2? Doesn't sound like it. That's extremely basic networking to know that wouldn't work. Your Pi address with the port should work if it's listening on all addresses. Probably should just set it to listen on everything like 0.0.0.0:2342 or whatever
different IP address in docker
yes that's docker's virtual network, not related to your "home network" and it isn't outside of it. It's a virtual network internal of docker which is inside of your pi which is inside of your home network
1 reply
if photoprism is only listening for localhost specifically and not anything address that's why it didn't respond to your pi's ip
oh I assumed pi2 from the tutorial you linked.
what you said
I assume can be set to 192.168.x.x:2342 whatever your pi's ip is
what you said
PHOTOPRISM_SITE_URL https://docs.photoprism.org/getting-started/config-options/I assume can be set to 192.168.x.x:2342 whatever your pi's ip is
1 reply
no idea, you can try it. Normally setting 0.0.0.0:port on anything binds it to that port and every address on the machine
not sure about photoprism specifically
So I just used this command (
docker-compose logs --tail=25 -f) to get the logs and it seems like something is going wrong. This is popping up every second or so in the logs: mariadb | 2021-06-18 16:15:02 141 [Warning] Access denied for user 'photoprism'@'172.18.0.3' (using password: YES)
1 reply
For some reason the logs are also two hours behind, even though I set the timezone via
sudo raspi-config. So either it chose to ignore the configured timezone or the logs really are two hours behind.
Sounds like logs are just always on utc. Which in many cases is what you want for application logging. Makes parsing easier.
1 reply