Alec's Projects/Lobby

Discussion/help for any of my projects, such as gometalinter, kingpin, voluptuous, EntityX, etc.

Naveen Nathan

@nnathan

I'm having a curious problem with pytest-voluptuous==1.1.0 (and voluptuous==0.11.5). Below timing.1.source has value "create," and the actual value is "create," but get assertion error and this doesn't really compute?

I should point out that timing.0 is fine. timing.1.duration_in_ms is also fine. Just timing.1.source that's failing.

v1_base_url = 'http://127.0.0.1:64387/v1'

    def test_customers_create(v1_base_url):
        url = v1_base_url + "/customers"

        data = {
            "first_name": "john",
            "last_name": "doe",
            "email": "johndoe@example.com",
            "phone_number": "04555555555"
        }

        r = requests.post(url, json=data)

        assert r.status_code == 200
>       assert r.json() == S(
            {
                "success": True,
                "timing": [
                    {"duration_in_ms": int, "source": "backend_api_call"},
                    {"duration_in_ms": int, "source": "create"}
                ],
                "messages": ["Customer created", "OK"],
                "response": {"unnecessary": "crap"}
        )
E       AssertionError: assert failed to validation error(s):
E         - timing.1.source: not a valid value (actual: 'create')

Naveen Nathan

@nnathan

oh nevermind, it's all good, i met this homeless bum on the street - he was an expert in pytest-voluptuous, told me that the slice in timing when specified in the S() is actually calling the underlying Schema voluptuous function and that slices have a very special meaning - that is, that the first match is tried and if it fails it fails and doesn't backtrac, but there's some kind of LiteralArray thing - anyway, I gave him a few dollars to spend a night at a hotel and have a shower - real good guy.

Alec Thomas

@alecthomas

@nnathan that's good to hear!

i think the validator you're looking for is ExactSequence

fwiw :)

Naveen Nathan

@nnathan

Ok I have a better understanding now of how a basic list works - it's just a list of unordered valid inputs - so ['a', 'b'] in a schema will accept ['b']*100. So it will not validate against the 'a', but will validate against the 'b' for each item in the list. Now going based on my above example - I still don't know why it failed the assertion (I get I want an ExactSequence and not a list of objects that are valid inputs), but i'm pretty sure it was receiving a valid input - and that the input - specifically the timing section, should've matched the valid input list for the timings in the schema above:

{
  "success": true,
  "response": { "unnecessary": "crap" },
  "timing": [
    {
      "duration_in_ms": 629,
      "source": "backend_api_call"
    },
    {
      "duration_in_ms": 655,
      "source": "create"
    }
  ],
  "messages": [
    "Customer created",
    "OK"
  ],
  "time": "2019-08-12T03:53:41.070455Z"
}

Alec Thomas

@alecthomas

no, it will also validate b

it failed because {...} matched, but then failed on a sub-key

voluptuous will not backtrack, so once it enters a sub-schema (ie. {...}) it will not exit

Naveen Nathan

@nnathan

The brain is ticking again. I get it -- had to read the (lack of) backtrack explanation in README, but makes sense now.

Ryan Gonzalez

@refi64

Question,

Is there a way to have posix-style parsing, where option parsing stops after the first argument?

E.g. if I run myapp -a b -c, b -c are non-options

I'm using kingpin for a Go tool since it's pretty much the sanest Go option parsing library by a long shot, but I really need to support this...

Alec Thomas

@alecthomas

No that is not supported. Kingpin supports -- to separate remaining args.

Mmm actually there is an interspersed option that I forgot about

Ryan Gonzalez

@refi64

Oof that sucks...would there be e.g. a PR accepted to add this somehow? GLib's GOptionContext calls it "struct posix mode", so maybe a .StrictPosix() on a Command instance?

Alec Thomas

@alecthomas

Did you try interspersed?

Ryan Gonzalez

@refi64

Ooo that might actually be what I need, I'll try it tomorrow

Ryan Gonzalez

@refi64

Sorry for all the noise, but out of curiosity was there a reason alecthomas/kingpin#245 was never merged?

Nitish Malhotra

@nitishm

@alecthomas How can I Capture Time formatted values using participle. I am trying to capture [2016-04-15T20:17:00.310Z]. Not sure how to proceed ?!

Alec Thomas

@alecthomas

@nitishm if you're using the regex lexer (for example) you'll need to match and emit a token. eg. (?P<Time>\[\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d{3}Z\])

then you'll need to implement the participle.Capture interface on a custom type

eg. something like this

type Time time.Time

func (t *Time) Capture(tokens []string) error {
  s, err := time.Parse("...", tokens[0])
  ...
  *t = s
  return nil
}

Alec Thomas

@alecthomas

you could also omit the second step and just capture the string

Nitish Malhotra

@nitishm

Yup I ended up creating a lexer.Regexp() for this. It's a fascinating tool. Am still trying to wrap my head around it 😂. But every iteration it seems to make more sense.

I am trying to write a custom log parser for envoy access logs.

Alec Thomas

@alecthomas

/all I'm moving this to the Gopher's Slack, as Gitter is effectively dead/unmaintained.

@/all I'm moving this to the Gopher's Slack, as Gitter is effectively dead/unmaintained.

https://gophers.slack.com/messages/CN9DS8YF3

Tariq Ibrahim

@tariq1890

Gitter effectively dead?

Wha?

LittleMoi

@LittleMoi

Hey guys.Is there any book recommended for me to understand ECS?

FruitPunch5amurai

@FruitPunch5amurai

Hey , i need some assistance with an issue im having. For EntityX, i have a system thats recieving ComponentRemoved Event. However, when i destroy an entity, it doesnt seem like the system is recieving the ComponentRemovedEvent.

The component is attached to the entity

thestics

@thestics

Hello there. I have a question about voluptuous. Is there any way to restrict a name of a field
something like:

schema = Schema({
    lambda name: '.' in name: str
})

Aurélien Gasser

@AurelienGasser

Hi Alec, thanks a lot for your Go serializer benchmark!
I'm trying to compress/decompress data - that data will only be used by my Go program.
I'm having a hard time understanding the "Totals" section of https://github.com/alecthomas/go_serialization_benchmarks. What do the columns mean? In particular, I'm interested in: speed of compression / speed of decompression / compressed payload size. Which columns should I look at ? How can I compare the profobuf row with the gob row?
Thank you!

Michael Pfaff

@mpfaff

I'm interested in something similar to Aurélien, but I have a slight focus on deserialization speed

thomas

@yaythomas

re chroma: hi Alec, any chance of a version bump on chroma so can hugo can pick up the latest?

Иван Сердюк

@oceanfish81_twitter

Hi. Got such a bug:

$ go build -i -v -x
WORK=/tmp/go-build822795550
mkdir -p $WORK/b017/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/alecthomas/
cp /home/oceanfish81/.cache/go-build/3f/3f17f09c67a788726b59d58fcca2958f8e89f05f49c82be2d1c6e75183bf139a-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/alecthomas/libunits.a
mkdir -p $WORK/b020/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/
cp /home/oceanfish81/.cache/go-build/1d/1d82d5034089a023b328cb96570d592006677007d9489fa77d5a6a64c584b247-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/liblanguage.a
rm -r $WORK/b017/
rm -r $WORK/b020/
mkdir -p $WORK/b022/
cp /home/oceanfish81/.cache/go-build/00/007e3ade50871872acfa250f6dcd292a0f846c61c8739ab18f79bccfcea98082-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/libtranslation.a
rm -r $WORK/b022/
mkdir -p $WORK/b025/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pelletier/
cp /home/oceanfish81/.cache/go-build/1e/1e1e4b85589ef24d39f9fc768b9bc7f8719ca407fca7c59fe6f650ae7bc96d43-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pelletier/libgo-toml.a
mkdir -p $WORK/b035/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/
cp /home/oceanfish81/.cache/go-build/17/17b6e6437480a8d8489dcaf96914181618b1641928828f8227fa59e5c2b5fc0b-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/libyaml.v2.a
rm -r $WORK/b025/
rm -r $WORK/b035/
mkdir -p $WORK/b019/
cp /home/oceanfish81/.cache/go-build/cf/cf0ac62a87259aadf849489ca255fad6bf03bf9a1a31bb4df5dcf87ce8232e45-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/libbundle.a
rm -r $WORK/b019/
mkdir -p $WORK/b018/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/
cp /home/oceanfish81/.cache/go-build/26/267db29519d8a5d352c042db18c08aa7a0f3e408c4b8720cc52aa9195c0951ab-d /home/oceanfish81/go/pkg/gccgo_linuxamd64/github.com/nicksnyder/go-i18n/libi18n.a
rm -r $WORK/b018/ /home/oceanfish81/go_projects/gometalinter
mkdir -p $WORK/b001/
cd $WORK
/home/oceanfish81/gollvm_dist/bin/llvm-goc -fgo-importcfg=/dev/null -c -x c - -o /dev/null || true
mkdir -p $WORK/b001/importcfgroot/github.com/google
ln -s /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/google/libshlex.a $WORK/b001/importcfgroot/github.com/google/libshlex.a
mkdir -p $WORK/b001/importcfgroot/gopkg.in/alecthomas
ln -s /home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/alecthomas/libkingpin.v3-unstable.a $WORK/b001/importcfgroot/gopkg.in/alecthomas/libkingpin.v3-unstable.a
cd /home/oceanfish81/go_projects/gometalinter
/home/oceanfish81/gollvmdist/bin/llvm-goc -c -O2 -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -fgo-relative-import-path=/home/oceanfish81/go_projects/gometalinter -o $WORK/b001/go.o -I $WORK/b001/importcfgroot ./aggregate.go ./checkstyle.go ./config.go ./directives.go ./execute.go ./issue.go ./linters.go ./main.go ./partition.go ./stringset.go

_/home/oceanfish81/go_projects/gometalinter

./main.go:36:95: error: argument 1 has incompatible type (different parameter types)
./main.go:38:101: error: argument 1 has incompatible type (different parameter types)
./main.go:39:100: error: argument 1 has incompatible type (different parameter types)
./main.go:40:84: error: argument 1 has incompatible type (different parameter types)
./main.go:43:57: error: argument 1 has incompatible type (different parameter types)
./main.go:44:55: error: argument 1 has incompatible type (different parameter types)
./main.go:200:13: error: argument 1 has incompatible type (different parameter types)

vellotis

@vellotis

@ https://github.com/alecthomas/participle

I appreciate the work on github.com/alecthomas/participle library :thumbsup: !

I'm writing a dynamic parser/lexer for EBNF grammar. Currently, I'm trying to use Go EBNF grammar that I derived from Go specification HTML (https://gist.github.com/vellotis/d59980f924309e0258222f26c02218fb). For parsing EBNF grammar I used Go's https://github.com/golang/exp/blob/master/ebnf/parser.go plus a bit of instrumentation of the grammar productions so it would fulfill my needs.
I have almost finished with my 1st working version of the lexer as well. But with a few complications with the Go's EBNF grammar. Probably the problem is how I handle which of the EBNF alternations should be resolved if multiple of them return a value.

Example:

Go's short variable declaration as input
simple := "value"

        type testableShortVarDecl struct {
            ShortVarDecl `@@ term`
        }

        // Collect instrumented version of github.com/golang/exp/ebnf/parser.go returned grammar
        grammar := GoGrammar()
        // Putting together my lexer with a starting production of "StatementList"
        lexerDefinition, err := NewLexer(grammar, ebnf.InitialProduction("StatementList"))
        require.NoError(t, err)

        // Build a participle parser
        parser, err := participle.Build(&testableShortVarDecl{},
            participle.Lexer(lexerDefinition),
            participle.Elide("ignored"))
        require.NoError(t, err)

        shortVarDecl := &testableShortVarDecl{}

        err = parser.ParseString("", `single := "value"`, shortVarDecl)

        assert.NoError(t, err)

The problem is that for production "SimpleStmt" it finds a matching lexical token for ExpressionStmt -> Expression -> UnaryExpr -> PrimaryExpr -> Operand -> OperandName -> identifier(= "simple") and resolves the whole alternation and doesn't reach to ShortVarDecl at all.

StatementList = { Statement ";" } .
Statement =
Declaration | LabeledStmt | SimpleStmt |
GoStmt | ReturnStmt | BreakStmt | ContinueStmt | GotoStmt |
FallthroughStmt | Block | IfStmt | SwitchStmt | SelectStmt | ForStmt |
DeferStmt .
SimpleStmt = EmptyStmt | ExpressionStmt | SendStmt | IncDecStmt | Assignment | ShortVarDecl .
ExpressionStmt = Expression .
Expression = UnaryExpr | Expression binary_op Expression .
UnaryExpr = PrimaryExpr | unary_op UnaryExpr .
PrimaryExpr =
Operand |
Conversion |
MethodExpr |
PrimaryExpr Selector |
PrimaryExpr Index |
PrimaryExpr Slice |
PrimaryExpr TypeAssertion |
PrimaryExpr Arguments .
Operand = Literal | OperandName | "(" Expression ")" .
OperandName = identifier | QualifiedIdent .
= "simple"

Does any of you have any expertise in EBNF and picking a resolving alternate element when many of them resolve?
Or could you point me to another discussion board? I know my last option is StackOverflow, but I would like to have a discussion, not just a solution at first.

I have currently implemented the alternation resolving as a sequential matching of the elements. The first one that matches will be selected.

vellotis

@vellotis

I know that the EBNF specification states that there is no specific order for how the alternation elements matching should be visited. But how to understand it in terms of the selection of the true matching element?

vellotis

@vellotis

Discussing by myself. I probably need some Backtracking

http://ceur-ws.org/Vol-928/0324.pdf#page=3

Backtracking occurs in the Sequence expression. If e1 succeeds and consumes
some input, and then e2 fails, the cursor is moved back to where it was before
trying e1. If this Sequence was called as the first alternative in a Choice, Choice
has an opportunity to try another alternative on the same input. However, the
backtracking is limited: once e1 in the Choice e1|e2 succeeded, e2 will never be
tried on the same input, even if the parse fails later on.

vellotis

@vellotis

Rubber ducking worked! Thanks! :smile:

xanderle

@xanderle_gitlab

trying to use participle to parse fluent bit config files.
im using https://github.com/calyptia/go-fluentbit-config/blob/master/parser.go as a base and
an example input file is

[SERVICE]
    Flush     5
    Daemon    off
    Log_Level debug

[INPUT]
    Name  cpu
    Tag   my_cpu

[OUTPUT]
    Name  stdout
    Match my*cpu

I dont want as many complex "value" fields. i just want to parse everything as a string and I can't work out how to do it.

for instances i can have ${PATH} or <tags> and i just want to parse it as a string

xanderle

@xanderle_gitlab

i guess what im really asking is how do i parse

[NAME]
  key value

[Name2]
  key value
  key2 value2

Where communities thrive

People

Repo info

Activity

_/home/oceanfish81/go_projects/gometalinter