Alec's Projects/Lobby

Discussion/help for any of my projects, such as gometalinter, kingpin, voluptuous, EntityX, etc.

tmartiro

@tmartiro

Could you please help to group same section
Input file

iface eth0
 address 1.1.1.1/32

iface eth0
 address 2.2.2.2/32

I want to get such output

{
    "iface" : "eth0",
    "addresses" : [
        {"address" : "1.1.1.1/32" },
        {"address" : "2.2.2.2/32" }
    ]
}

Is it possible to do it in parser, or I have to mutate the structures after the parser finishes its job in order to achieve such kind of output ?

Alec Thomas

@alecthomas

You’ll have to do that in a post-process

Or a method on the parent node to transform it into that form on the fly

tmartiro

@tmartiro

I am interested in the method on parent node. Do you have any example related to it ?

vito-c

@vito-c

how do I create a custom style for chroma? or port over an existing vim style :)

vito-c

@vito-c

derp i got it :D

Naveen Nathan

@nnathan

I'm having a curious problem with pytest-voluptuous==1.1.0 (and voluptuous==0.11.5). Below timing.1.source has value "create," and the actual value is "create," but get assertion error and this doesn't really compute?

I should point out that timing.0 is fine. timing.1.duration_in_ms is also fine. Just timing.1.source that's failing.

v1_base_url = 'http://127.0.0.1:64387/v1'

    def test_customers_create(v1_base_url):
        url = v1_base_url + "/customers"

        data = {
            "first_name": "john",
            "last_name": "doe",
            "email": "johndoe@example.com",
            "phone_number": "04555555555"
        }

        r = requests.post(url, json=data)

        assert r.status_code == 200
>       assert r.json() == S(
            {
                "success": True,
                "timing": [
                    {"duration_in_ms": int, "source": "backend_api_call"},
                    {"duration_in_ms": int, "source": "create"}
                ],
                "messages": ["Customer created", "OK"],
                "response": {"unnecessary": "crap"}
        )
E       AssertionError: assert failed to validation error(s):
E         - timing.1.source: not a valid value (actual: 'create')

Naveen Nathan

@nnathan

oh nevermind, it's all good, i met this homeless bum on the street - he was an expert in pytest-voluptuous, told me that the slice in timing when specified in the S() is actually calling the underlying Schema voluptuous function and that slices have a very special meaning - that is, that the first match is tried and if it fails it fails and doesn't backtrac, but there's some kind of LiteralArray thing - anyway, I gave him a few dollars to spend a night at a hotel and have a shower - real good guy.

Alec Thomas

@alecthomas

@nnathan that's good to hear!

i think the validator you're looking for is ExactSequence

fwiw :)

Naveen Nathan

@nnathan

Ok I have a better understanding now of how a basic list works - it's just a list of unordered valid inputs - so ['a', 'b'] in a schema will accept ['b']*100. So it will not validate against the 'a', but will validate against the 'b' for each item in the list. Now going based on my above example - I still don't know why it failed the assertion (I get I want an ExactSequence and not a list of objects that are valid inputs), but i'm pretty sure it was receiving a valid input - and that the input - specifically the timing section, should've matched the valid input list for the timings in the schema above:

{
  "success": true,
  "response": { "unnecessary": "crap" },
  "timing": [
    {
      "duration_in_ms": 629,
      "source": "backend_api_call"
    },
    {
      "duration_in_ms": 655,
      "source": "create"
    }
  ],
  "messages": [
    "Customer created",
    "OK"
  ],
  "time": "2019-08-12T03:53:41.070455Z"
}

Alec Thomas

@alecthomas

no, it will also validate b

it failed because {...} matched, but then failed on a sub-key

voluptuous will not backtrack, so once it enters a sub-schema (ie. {...}) it will not exit

Naveen Nathan

@nnathan

The brain is ticking again. I get it -- had to read the (lack of) backtrack explanation in README, but makes sense now.

Ryan Gonzalez

@refi64

Question,

Is there a way to have posix-style parsing, where option parsing stops after the first argument?

E.g. if I run myapp -a b -c, b -c are non-options

I'm using kingpin for a Go tool since it's pretty much the sanest Go option parsing library by a long shot, but I really need to support this...

Alec Thomas

@alecthomas

No that is not supported. Kingpin supports -- to separate remaining args.

Mmm actually there is an interspersed option that I forgot about

Ryan Gonzalez

@refi64

Oof that sucks...would there be e.g. a PR accepted to add this somehow? GLib's GOptionContext calls it "struct posix mode", so maybe a .StrictPosix() on a Command instance?

Alec Thomas

@alecthomas

Did you try interspersed?

Ryan Gonzalez

@refi64

Ooo that might actually be what I need, I'll try it tomorrow

Ryan Gonzalez

@refi64

Sorry for all the noise, but out of curiosity was there a reason alecthomas/kingpin#245 was never merged?

Nitish Malhotra

@nitishm

@alecthomas How can I Capture Time formatted values using participle. I am trying to capture [2016-04-15T20:17:00.310Z]. Not sure how to proceed ?!

Alec Thomas

@alecthomas

@nitishm if you're using the regex lexer (for example) you'll need to match and emit a token. eg. (?P<Time>\[\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d\.\d{3}Z\])

then you'll need to implement the participle.Capture interface on a custom type

eg. something like this

type Time time.Time

func (t *Time) Capture(tokens []string) error {
  s, err := time.Parse("...", tokens[0])
  ...
  *t = s
  return nil
}

Alec Thomas

@alecthomas

you could also omit the second step and just capture the string

Nitish Malhotra

@nitishm

Yup I ended up creating a lexer.Regexp() for this. It's a fascinating tool. Am still trying to wrap my head around it 😂. But every iteration it seems to make more sense.

I am trying to write a custom log parser for envoy access logs.

Alec Thomas

@alecthomas

/all I'm moving this to the Gopher's Slack, as Gitter is effectively dead/unmaintained.

@/all I'm moving this to the Gopher's Slack, as Gitter is effectively dead/unmaintained.

https://gophers.slack.com/messages/CN9DS8YF3

Tariq Ibrahim

@tariq1890

Gitter effectively dead?

Wha?

LittleMoi

@LittleMoi

Hey guys.Is there any book recommended for me to understand ECS?

FruitPunch5amurai

@FruitPunch5amurai

Hey , i need some assistance with an issue im having. For EntityX, i have a system thats recieving ComponentRemoved Event. However, when i destroy an entity, it doesnt seem like the system is recieving the ComponentRemovedEvent.

The component is attached to the entity

thestics

@thestics

Hello there. I have a question about voluptuous. Is there any way to restrict a name of a field
something like:

schema = Schema({
    lambda name: '.' in name: str
})

Aurélien Gasser

@AurelienGasser

Hi Alec, thanks a lot for your Go serializer benchmark!
I'm trying to compress/decompress data - that data will only be used by my Go program.
I'm having a hard time understanding the "Totals" section of https://github.com/alecthomas/go_serialization_benchmarks. What do the columns mean? In particular, I'm interested in: speed of compression / speed of decompression / compressed payload size. Which columns should I look at ? How can I compare the profobuf row with the gob row?
Thank you!

Michael Pfaff

@mpfaff

I'm interested in something similar to Aurélien, but I have a slight focus on deserialization speed

thomas

@yaythomas

re chroma: hi Alec, any chance of a version bump on chroma so can hugo can pick up the latest?

Иван Сердюк

@oceanfish81_twitter

Hi. Got such a bug:

$ go build -i -v -x
WORK=/tmp/go-build822795550
mkdir -p $WORK/b017/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/alecthomas/
cp /home/oceanfish81/.cache/go-build/3f/3f17f09c67a788726b59d58fcca2958f8e89f05f49c82be2d1c6e75183bf139a-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/alecthomas/libunits.a
mkdir -p $WORK/b020/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/
cp /home/oceanfish81/.cache/go-build/1d/1d82d5034089a023b328cb96570d592006677007d9489fa77d5a6a64c584b247-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/liblanguage.a
rm -r $WORK/b017/
rm -r $WORK/b020/
mkdir -p $WORK/b022/
cp /home/oceanfish81/.cache/go-build/00/007e3ade50871872acfa250f6dcd292a0f846c61c8739ab18f79bccfcea98082-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/libtranslation.a
rm -r $WORK/b022/
mkdir -p $WORK/b025/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pelletier/
cp /home/oceanfish81/.cache/go-build/1e/1e1e4b85589ef24d39f9fc768b9bc7f8719ca407fca7c59fe6f650ae7bc96d43-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/pelletier/libgo-toml.a
mkdir -p $WORK/b035/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/
cp /home/oceanfish81/.cache/go-build/17/17b6e6437480a8d8489dcaf96914181618b1641928828f8227fa59e5c2b5fc0b-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/libyaml.v2.a
rm -r $WORK/b025/
rm -r $WORK/b035/
mkdir -p $WORK/b019/
cp /home/oceanfish81/.cache/go-build/cf/cf0ac62a87259aadf849489ca255fad6bf03bf9a1a31bb4df5dcf87ce8232e45-d /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/i18n/libbundle.a
rm -r $WORK/b019/
mkdir -p $WORK/b018/
mkdir -p /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/nicksnyder/go-i18n/
cp /home/oceanfish81/.cache/go-build/26/267db29519d8a5d352c042db18c08aa7a0f3e408c4b8720cc52aa9195c0951ab-d /home/oceanfish81/go/pkg/gccgo_linuxamd64/github.com/nicksnyder/go-i18n/libi18n.a
rm -r $WORK/b018/ /home/oceanfish81/go_projects/gometalinter
mkdir -p $WORK/b001/
cd $WORK
/home/oceanfish81/gollvm_dist/bin/llvm-goc -fgo-importcfg=/dev/null -c -x c - -o /dev/null || true
mkdir -p $WORK/b001/importcfgroot/github.com/google
ln -s /home/oceanfish81/go/pkg/gccgo_linux_amd64/github.com/google/libshlex.a $WORK/b001/importcfgroot/github.com/google/libshlex.a
mkdir -p $WORK/b001/importcfgroot/gopkg.in/alecthomas
ln -s /home/oceanfish81/go/pkg/gccgo_linux_amd64/gopkg.in/alecthomas/libkingpin.v3-unstable.a $WORK/b001/importcfgroot/gopkg.in/alecthomas/libkingpin.v3-unstable.a
cd /home/oceanfish81/go_projects/gometalinter
/home/oceanfish81/gollvmdist/bin/llvm-goc -c -O2 -g -m64 -fdebug-prefix-map=$WORK=/tmp/go-build -gno-record-gcc-switches -fgo-relative-import-path=/home/oceanfish81/go_projects/gometalinter -o $WORK/b001/go.o -I $WORK/b001/importcfgroot ./aggregate.go ./checkstyle.go ./config.go ./directives.go ./execute.go ./issue.go ./linters.go ./main.go ./partition.go ./stringset.go

_/home/oceanfish81/go_projects/gometalinter

./main.go:36:95: error: argument 1 has incompatible type (different parameter types)
./main.go:38:101: error: argument 1 has incompatible type (different parameter types)
./main.go:39:100: error: argument 1 has incompatible type (different parameter types)
./main.go:40:84: error: argument 1 has incompatible type (different parameter types)
./main.go:43:57: error: argument 1 has incompatible type (different parameter types)
./main.go:44:55: error: argument 1 has incompatible type (different parameter types)
./main.go:200:13: error: argument 1 has incompatible type (different parameter types)

vellotis

@vellotis

@ https://github.com/alecthomas/participle

I appreciate the work on github.com/alecthomas/participle library :thumbsup: !

I'm writing a dynamic parser/lexer for EBNF grammar. Currently, I'm trying to use Go EBNF grammar that I derived from Go specification HTML (https://gist.github.com/vellotis/d59980f924309e0258222f26c02218fb). For parsing EBNF grammar I used Go's https://github.com/golang/exp/blob/master/ebnf/parser.go plus a bit of instrumentation of the grammar productions so it would fulfill my needs.
I have almost finished with my 1st working version of the lexer as well. But with a few complications with the Go's EBNF grammar. Probably the problem is how I handle which of the EBNF alternations should be resolved if multiple of them return a value.

Example:

Go's short variable declaration as input
simple := "value"

        type testableShortVarDecl struct {
            ShortVarDecl `@@ term`
        }

        // Collect instrumented version of github.com/golang/exp/ebnf/parser.go returned grammar
        grammar := GoGrammar()
        // Putting together my lexer with a starting production of "StatementList"
        lexerDefinition, err := NewLexer(grammar, ebnf.InitialProduction("StatementList"))
        require.NoError(t, err)

        // Build a participle parser
        parser, err := participle.Build(&testableShortVarDecl{},
            participle.Lexer(lexerDefinition),
            participle.Elide("ignored"))
        require.NoError(t, err)

        shortVarDecl := &testableShortVarDecl{}

        err = parser.ParseString("", `single := "value"`, shortVarDecl)

        assert.NoError(t, err)

The problem is that for production "SimpleStmt" it finds a matching lexical token for ExpressionStmt -> Expression -> UnaryExpr -> PrimaryExpr -> Operand -> OperandName -> identifier(= "simple") and resolves the whole alternation and doesn't reach to ShortVarDecl at all.

StatementList = { Statement ";" } .
Statement =
Declaration | LabeledStmt | SimpleStmt |
GoStmt | ReturnStmt | BreakStmt | ContinueStmt | GotoStmt |
FallthroughStmt | Block | IfStmt | SwitchStmt | SelectStmt | ForStmt |
DeferStmt .
SimpleStmt = EmptyStmt | ExpressionStmt | SendStmt | IncDecStmt | Assignment | ShortVarDecl .
ExpressionStmt = Expression .
Expression = UnaryExpr | Expression binary_op Expression .
UnaryExpr = PrimaryExpr | unary_op UnaryExpr .
PrimaryExpr =
Operand |
Conversion |
MethodExpr |
PrimaryExpr Selector |
PrimaryExpr Index |
PrimaryExpr Slice |
PrimaryExpr TypeAssertion |
PrimaryExpr Arguments .
Operand = Literal | OperandName | "(" Expression ")" .
OperandName = identifier | QualifiedIdent .
= "simple"

Does any of you have any expertise in EBNF and picking a resolving alternate element when many of them resolve?
Or could you point me to another discussion board? I know my last option is StackOverflow, but I would like to have a discussion, not just a solution at first.

I have currently implemented the alternation resolving as a sequential matching of the elements. The first one that matches will be selected.

Where communities thrive

People

Repo info

Activity

_/home/oceanfish81/go_projects/gometalinter