ovh/container-registry

All about the container registry product!

Colin GILLE

@Congelli501

Thanks @DevOps-TauRes !

Baboon

@LouisKottmann

Thanks

Thomas Garcia

@tomgarcia__twitter

Hi All, we are aware about your troubles. we have problems with the storage. You can follow this issue here: http://travaux.ovh.net/?do=details&id=47499&

MoJoce

@MoJoce

Hello, same pb here...

DevOps at TauRes

@DevOps-TauRes

@tomgarcia__twitter Are there any updates on this? Another workday has started and our developers are in deep trouble here.

Simon Guyennet

@sguyennet

Hi @DevOps-TauRes The storage issue was just updated:
http://travaux.ovh.net/?do=details&id=47481

Baboon

@LouisKottmann

it's just a scaling issue?

Baboon

@LouisKottmann

the issue is still ongoing, all the pushes failed this morning...

MoJoce

@MoJoce

me too

Simon Guyennet

@sguyennet

Hi,
The storage issue is currently still in investigation. Please follow the progess here:
http://travaux.ovh.net/?do=details&id=47481

Colin GILLE

@Congelli501

I confirm, push is failing again on my repo too

Colin GILLE

@Congelli501

Our registry is still failing randomly, any ETA on a definitive fix?
We are losing a lot of time retrying pipelines.

Jérémie MONSINJON

@jMonsinjon

I understand and I'm really sorry...
But no ETA from team in charge of it...

Treussart Matthieu

@treussart

Hi, how can I join the notary server ? I try this : notary -s https://[id].gra7.container-registry.ovh.net/notary -d ~/.docker/trust list https://[id].gra7.container-registry.ovh.net/devops/base but I have the Go error : invalid character '<' looking for beginning of value

10 replies

Guilhem Bonnefille

@gbonnefille

Hi all, we are regularly experriencing a random push error. We are using Kaniko and the message is not really useful to diagnose:

error pushing image: failed to push to destination xxx.gra5.container-registry.ovh.net/project/repo:tag: PUT https://xxx.gra5.container-registry.ovh.net/v2/repo/repo/blobs/uploads/a28bfac9-a522-4249-8a11-b7c8341ce79a?_state=REDACTED&digest=sha256%3A33df0b4b677132b4f20990e1443b5b8b809f2d3bf3ce22fd3c7a2ec5a36623bb: UNKNOWN: unknown error; map[DriverName:s3aws Path:/docker/registry/v2/repositories/project/repo/_upload/a28bfac9-a522-4249-8a11-b7c8341ce79a/data]

Is it related to our usage? Does Kaniko is not safe here? I saw there is other issues related to storage. Can we consider such issue is related to global infrastructure?

Jérémy Levilain

@IamBlueSlime

Same issue here with a classic Docker push. This issue is annoying in automated deployments...

Jérémie MONSINJON

@jMonsinjon

@gbonnefille @IamBlueSlime
Everything's seems OK on our side...
Are you still in trouble? Was it a temporary downtime?

5 replies

DevOps at TauRes

@DevOps-TauRes

Hi, is there an update to the timeline when the service is upgraded to Harbor 2.0 or even better 2.1?

3 replies

DevOps at TauRes

@DevOps-TauRes

The Garbage Collection of our Harbor instance seems to be broken, "Internal Server Error". Can somebody help us?

1 reply

DevOps at TauRes

@DevOps-TauRes

We are getting more and more error when it comes to pushing images, we need help!

error pushing image: failed to push to destination XXX.XXX.container-registry.ovh.net/library/XXX-XXX:rkYuxab: Head "https://storage.gra.cloud.ovh.net/XXX-d697-4588-bfda-eb24eaa98aa6/files/docker/registry/v2/blobs/sha256/f2/f26661f4bd05f0baedb0825bd32a7704570e64aa6f587ff7e343e8f62d4270c5/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=18677960e2d84ec482a66615d254962a%2F20210114%2FGRA%2Fs3%2Faws4_request&X-Amz-Date=20210114T080021Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=XXX": read tcp 10.42.208.30:46442->54.38.230.75:443: read: connection reset by peer

vkukk

@vkukk

Registry service broken again? I can't delete old tags again.

1 reply

Andy Tan

@tanandy

Hi guys, i just created a simple container registry from scratch and got Error status

3 replies

DevOps at TauRes

@DevOps-TauRes

We are experiencing issues with the storage when we're pushing images, we have an open ticket for this: 1516595

PUT https://XXX.gra7.container-registry.ovh.net/v2/library/tools-XXX/blobs/uploads/72f8e4f6-6c04-4022-9a82-51f91ac29cdd?_state=REDACTED&digest=sha256%3A1064dab604aaf3f83a795daf993808926d6c578bee4ebbc16cbfb86eb4c18129: UNKNOWN: unknown error; map[DriverName:s3aws Path:/docker/registry/v2/repositories/library/tools-XXX/_uploads/72f8e4f6-6c04-4022-9a82-51f91ac29cdd/data]

Jérémy Levilain

@IamBlueSlime

I have a huge number of time the unknown blob error while pushing my images. This cause my deploy pipeline to fail. If this continue I will have to leave the OVH Docker Registry cause it's just not stable enough to use in production. My pipelines are long and see them fail after 15 minutes is annoying.. I've already posted some messages here for the same issue (nov. 18 2020). Do you have isolated the issue on your side? What could you do to prevent this in the future?

Andy Tan

@tanandy

hi guys, can we plug sso on harbor (DEX)? (or do you plan to support it soon ?) we would like to handle users with external oidc providers like github. Having, only local database is a problem since we want to handle users rights in a common tool

1 reply

Arthur Lutz

@arthurlogilab_gitlab

I've been told this could be a place to get support for s3/object storage ACLs... it seems this room is about the container (docker) product at OVH, am I correct ?

5 replies

tsn77130

@tsn77130

hi guys, we've got a problem with private registry, which is no more accessible (error pulling image configuration: error parsing HTTP 400 response body: invalid character '<' looking for beginning of value: "<html><h1>Bad Request</h1><p>The server could not comply with the request since it is either malformed or otherwise incorrect.</p></html>").

could you please take a look ? (registry ID : xe1pyfka.gra5.container-registry.ovh.net)

1 reply

Arthur Lutz

@arthurlogilab_gitlab

hello, is anyone from ovh here and can answer my question ?

Colin GILLE

@Congelli501

Hey, is there any progress on registry garbage collection support / fix ?

3 replies

Bmagic

@bmagic

Hello, when i want to delete a tag on my repo I have this error :

Error occurred when to handle request in immutable handler: failed to get immutable tag rule by projectID 3, error: pq: column t0.disabled does not exist

Anyone with the same error ?

1 reply

Waseem Awashra

@wawashra

hello why ovh Private Registry not work ?

Colin GILLE

@Congelli501

Hi, we have 500 errors on image push
received unexpected HTTP status: 500 Internal Server Error

1 reply

yctn

@yctn

i changed the Robot Token Expiration (Days) to like 3000 but exsisting robot account are not updated is this normal behavior?

is there any way i can prevent all the current robot accounts from expiring?

Wazabiii

@Wazabiii

Hi, is there any news about the upgrade to Harbor 2.x?

Jérémie MONSINJON

@jMonsinjon

Harbor 2 will be available in a couple of weeks

But we will not migrate actual registries until few month

Colin GILLE

@Congelli501

Hey, we got a network / push error:

$ docker push 1yvuiziu.gra5.container-registry.ovh.net/blg/client-service/accounting-sync-server:latest
The push refers to repository [1yvuiziu.gra5.container-registry.ovh.net/blg/client-service/accounting-sync-server]
739e98be7a16: Preparing
24d46fe38e8a: Preparing
02a9ca88eab1: Preparing
30e6cfc5d05b: Preparing
6169bf830ae6: Preparing
30e6cfc5d05b: Layer already exists
02a9ca88eab1: Layer already exists
24d46fe38e8a: Layer already exists
6169bf830ae6: Layer already exists
739e98be7a16: Pushed
blob upload unknown

1 reply

Colin GILLE

@Congelli501

Hi @xduthil_gitlab, retrying the push was enough to correct the issue and the pull works OK.
It's a problem for our CI system: each time we push into a deploy branch there is a chance the CI pipeline will fail because of the docker registry.
This is very time consuming for our dev team.

2 replies

Colin GILLE

@Congelli501

We have a very high rate of push failure since yesterday, nearly half our pipeline fails, could someone look into it ?

DBCL

@DB-CL

Hi, all my robots accounts cannot login anymore (401 unauthorized). Is there an issue or something ?

1 reply

Jérémy Levilain

@IamBlueSlime

Hi, I tried to upgrade my registry to the M-tier, now it is marked as errored and is not reachable anymore. Can someone from OVH can lookup into if possible? (PM for the registry link)

BarbeRousse

@mjoigny

Hi all, i get this message when i try to pull/push to my registry :

Error response from daemon: Get https://xe1pyfka.gra5.container-registry.ovh.net/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Is it related to some tasks here : http://travaux.ovh.net/?project=36&status=all&perpage=50

16 replies

Simon Guyennet

@sguyennet

Hi,
I just created an incident regarding this:
http://travaux.ovh.net/?do=details&id=49680
The team is on it.

1 reply

Jérémy Levilain

@IamBlueSlime

I have some issue with Harbor's timestamps, I was trying to setup retention policy to retain the images pushed or pulled in the last week but Harbor does not store correctly the last push and pull times (0001/01/01 00:00:00). Do someone know how to fix this?

|                                 Digest                                  |          Tag           | Kind  | Labels |     PushedTime      |     PulledTime      |     CreatedTime     | Retention |
|-------------------------------------------------------------------------|------------------------|-------|--------|---------------------|---------------------|---------------------|-----------|
| sha256:c2a2b6a793cdd1115e882555d342e8422da4353c60bd263c896f00ad877d0efb | dev                    | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2021/03/24 08:11:00 | RETAIN    |
| sha256:a52471350f3935cf25f23c124f3e674c1e20c7135e04aa4c449bcdc9e7e4ef49 | feat-xxx               | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2021/03/24 14:17:53 | DEL       |
| sha256:6d33ecc02aa4192cbf4942940d5998c9c14773d7de0a363de39e3979295d4a47 | latest                 | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2021/03/17 11:22:24 | RETAIN    |
| sha256:d89e25d943625c20be48fccc27c75685007fdaa96e50f1cbe4187da54ec0493a | staging                | image |        | 0001/01/01 00:00:00 | 0001/01/01 00:00:00 | 2021/03/24 09:02:30 | RETAIN    |

André Veelken

@ae-v

Yes. Same here: incorrect time stamps.

Our repository was set read-only several times. Why OVH?

Xavier Duthil

@xduthil_gitlab

Hello everyone, a DNS issue is currently impacting Managed Registries on GRA7: http://travaux.ovh.net/?do=details&id=49761

Fix is done, but DNS propagation is still in progress. A temporary workaround is described in the travaux.

1 reply

Wazabiii

@Wazabiii

Hi there,
I have an issue with my docker registry. When I push an image, layers are pushed but the manifest is not created.

docker push 9fc731v2.gra5.container-registry.ovh.net/docker-hub/drone/git:latest
The push refers to repository [9fc731v2.gra5.container-registry.ovh.net/docker-hub/drone/git]
5073ba39d063: Layer already exists 
62929d548adb: Layer already exists 
777b2c648970: Layer already exists 
latest: digest: sha256:4a030a46a38951a7ec39fbe0eaf924597218b67f4090ace3cdca9548d0a017a3 size: 948

docker pull 9fc731v2.gra5.container-registry.ovh.net/docker-hub/drone/git:latest
Error response from daemon: manifest for 9fc731v2.gra5.container-registry.ovh.net/docker-hub/drone/git:latest not found: manifest unknown: manifest unknown

When I try to retag the image with the harbor API I have this error: The system is in read only mode. Any modification is prohibited.

How can I solved the problem?
Thanks for the help

Where communities thrive

People

Repo info

Activity