by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 10:46
    php4umagento starred techdivision/import
  • May 28 08:18
    Hector68 starred techdivision/import
  • May 27 15:07

    wagnert on 23.x

    * Add #PAC-47 * Switch to techd… (compare)

  • May 27 15:06

    wagnert on 22.x

    (compare)

  • May 27 15:05

    wagnert on 22.x

    * Add #PAC-47 * Switch to techd… (compare)

  • May 27 14:59

    wagnert on 23.x

    * Add #PAC-47 * Switch to lates… (compare)

  • May 27 14:53

    wagnert on 22.x

    * Add #PAC-47 (compare)

  • May 27 14:49

    wagnert on 22.x

    Add #PAC-47 (compare)

  • May 26 16:49

    wagnert on 3.8.x

    Update composer dependencies (compare)

  • May 26 16:44

    wagnert on 22.x

    Update CHANGELOG.md (compare)

  • May 26 16:43

    wagnert on 22.x

    * Add #PAC-47 (compare)

  • May 26 16:40

    wagnert on 23.x

    * Add #PAC-47 * Switch to lates… (compare)

  • May 26 15:53

    wagnert on 22.x

    * Add #PAC-47 (compare)

  • May 25 14:15

    wagnert on 23.x

    * Add #PAC-47 * Switch to lates… (compare)

  • May 25 13:54

    wagnert on 22.x

    * Add #PAC-47 (compare)

  • May 22 13:21

    wagnert on 3.8.x

    * Add #PAC-46 * Add #PAC-96 (compare)

  • May 22 12:59

    wagnert on 22.x

    (compare)

  • May 22 12:58

    wagnert on 23.x

    (compare)

  • May 22 12:56

    wagnert on 22.x

    * Add #PAC-46 (compare)

  • May 22 12:52

    wagnert on 22.x

    * Add #PAC-46 (compare)

Rappels
@Rappels
@pathmissing Thanks for the reply. Yes, those are complete sets of data (for initial import, with all the data). I am trying to update price qty etc. If I try that using the normal web import with a csv file, it works fine with minimal columns and the url-key for example is not required this way, but the importer does require it.
Alexander Menk
@amenk
Hi :) I have technical question: How much of the Magento core importer does M2IF replace? Is it still based on it, or mostly working around it?
And another question: We have large XML files as an input from PIM system, XSLT transformations to flat XML files are already in place. Can I pass such data as an array to M2IF or do I have to write .CSV files?
Tim Wagner
@wagnert

@Rappels Up from version 3.8.x we don't use a centralized configuration file, instead we've switched to configuration snippets that has to be available in the subdirectory <magento-install-dir>/app/etc/configuration. To change the multiple field delimiter, simply add a snipped <magento-install-dir>/app/etc/configuration/csv-format.json with the content

{
  "multiple-field-delimiter": ";"
}

this should do the trick.

Hi @amenk, thanks for asking. M2IF doesn't use any code from the Magento core, but It provides the same functionality and supports the default Magento CSV format round about 99 %, so in general, there should not be too many changes in your CSV file to replace the default functionality with M2IF. Additionally M2IF comes with import functionality for attribute sets, attributes and categories, which allows you to import the complete catalog :-)
Alexander Menk
@amenk
@wagnert That is an ehm .. interesting approach. It sounds to me more like what MAGMI did in earlier times. I would feel better if the core code would be used, but I also know that Magento 2 core importer is more or less messy. Why was this approach chosen?
Rappels
@Rappels
@wagnert thanks for the info, i'll test it later today. Do you have anymore insights in to why the url-key seems to be required in the product imporT?
Alexander Menk
@amenk
@wagnert so does it directly write to the Database or use Magento Bulk API or something like that?
Tim Wagner
@wagnert
@amenk I don't know Magmi, but I've heard about it :) In general the Magento Import functionality that uses the core is very slow and memory intensive which makes it useless if someone needs to import a high amount of data in a minimum of time. So this is the reason why we decide not to use any core classes or the bulk import functionality. Yes, we're writing directly to the database because this is by far the fastest solution. As we provide M2IF and use in all of our projects meanwhile, we figured out, that the database doesn't had much changes which we have to be aware of, so the consequences have been taken into account. We're focusing on performance and a low memory profile and a maximum in flexibility, as M2IF is the base of our commercial solution called Pacemaker that provides a pipeline base approach to break down processes - like the import is - into the smallest possible unites to allow us coordination of processes e. g. with the indexers as well as a better monitoring and handling of each step. You don't need to specify a url-key, but you can. If not specified, the URL key will be prepared from the product name, but this can also be customized to your needs, in case you have custom requirements.
Alexander Menk
@amenk
@wagnert Thank you. But pacemaker is not needed to use M2IF, right?
Alexander Menk
@amenk
Funny part is that the Magento Core importer also tries to write quite directly to the database, as you can import wrong store_ids and so on (at least for customers)
Rappels
@Rappels
@wagnert thanks for the info. So M2IF kinda requires it? Would be easier to not require it since I want to keep the import as simple as possible when updating products and the normal import doesn't require it either.
Tim Wagner
@wagnert
@amenk No, but the next M2IF version will be renamed to Pacemaker Community ;-)
@Rappels Yes, if you want to change the multiple-field-delimiter, this has to be done with a configuration snipped. If you may create a feature request in techdivision/import-cli-simple repository, probably we can add a commandline option too :)
Rappels
@Rappels
@wagnert Hey Tim, thanks for the additional info. But my question is mainly, why do I need to give M2IF so much information in my CSV file to update just prices and stock, while the "normal" web import function in magento does NOT require that information (like url-key).
Tim Wagner
@wagnert
@Rappels You don't have to ... in case for the price import, you only have to specify a view columns as well as for the stock. You can invoke those imports with the dedicated command import:products:inventory + import:products:price as described here :-)
Rappels
@Rappels
@wagnert Thanks for your time! I'll take a look :)
Tim Wagner
@wagnert
@Rappels You're welcome :)
Alexander Menk
@amenk

And another question: We have large XML files as an input from PIM system, XSLT transformations to flat XML files are already in place. Can I pass such data as an array to M2IF or do I have to write .CSV files?

I think that question went a bit unnoticed :) Is the main Idea to write CSV files, or is there something like an array-input-adapter? oder even XML support?

Tim Wagner
@wagnert
Hi @amenk actually we only have a CSV file adapter, but the architecture will also allow to implement custom adapters e. g. for XML. If you’re intetested in that topic, we can discuss that 🙂
Alexander Menk
@amenk
@wagnert sure ... what are the pros/cons? I assume with CSVs it's a bit easier and more straight-forward?
Tim Wagner
@wagnert
@amenk CSV is, in any case, in our experience, the lowes common solution that nearly everybody can handle. XML is a bit more complex and so far, we didn't have the requirement to process XML files in one of our projects, yet. I personally would prefer XML, because it's better to validate, to manipulate, and to handle as well as there are more professional tools to deal with it. The big pro for CSV is definitely, that this is fully implemented, integrated, tested (for sure there can be bugs, but that's the way of life) and we'll use it in many projects. Therefore, if you're looking for the simplest and cost-efficient solution that can easily be integrated, I would recommend going the CSV way :-)
Alexander Menk
@amenk
@wagnert okay, first test (with attributes) went pretty well -- we have another specialty: The PIM export does not work fully with SKUs, but with PIM-IDs.. so after importing simple products we have to cross-reference the configurables via PIM IDs or look up the SKUs from the PIM IDs ... maybe the best solution would be to ask the PIM-exporter-team to include the SKUs as well :)
Tim Wagner
@wagnert
@amenk Yes, this would be, what I would recommend. Mapping the PIM IDs to SKUs during runtime with something like a lookup will probably slow down the import process. So I would try to avoid this and see if it'll be possible to get the already mapped PIM IDs with the import CSV.
Alexander Menk
@amenk
@wagnert Okay, will think about it. Thanks for the awesome support :)
Tim Wagner
@wagnert
@amenk You're welcome :) Sounds like an interesting project you're working on!!! Which PIM are you using?
Alexander Menk
@amenk
@wagnert Viamedici
Tim Wagner
@wagnert
All right :) So I'm curious to hear how things went on. Keep us updated!
Giuliano69
@Giuliano69

Dear Tim,
M2IF is a great tool for Magento2. I discover it searching on stack.exchange…. But what really surprise me is that it is still not so much speread around considering how much is powerful … hope it will get soon...

I found amazing the categories, attribute & attribute set import.
Hundreds of categories and attributes that should have been imported in Magento2 by hands…

Unfortunately I got also some problems with images import, and I am here to ask about it...

I wget import,simple,phar 3,8.12, and installed it m2root/bin
Using ubuntu 18,04 + PHP 7,2 + Mage2 2,3,5p1

By DESIGN the phar file try to import all the csv files from /var/importexport , without any json external json, nor any command line parameter.
The var/importexport directory must be created by hand, BUT this is VERY handy !!

BUT seems to me (hope to be wrong!) that you have NOT considered a default directory for image import. This is a great LIMITATION, considering that a similar feature exist for csv files…
PROPOSAL: could you consider to implement this default image directory in your next release ?

Tim Wagner
@wagnert
Hi @Giuliano69 Thanks for using M2IF 🙂 I’ll add this to the feature list, you’te riight, this is some kind of limitation 🙁
As it’s laye here, Ill have a detailed look at your problem tomorrow, hope this is ok for you. If possible, please provide s sample CSV and your configuration snippets, if you use some. Additionally a small instruction about the steps and commands you are invoking will help me to solve the problem quicker 🙂
Giuliano69
@Giuliano69
Thanks Tim for your reply. I opened an issue at
techdivision/import-cli-simple#247
giving the example files
I also wrote this night a second question about json config, but I cannot find it anymore. I'm a gitter chatter newby.. I will post it again with some hints for documentations...
Tim Wagner
@wagnert
@Giuliano69 Yes, I saw your issue and commented it ;-) Yes, for sure, post it again or create an issue on Github :)
Giuliano69
@Giuliano69

Thanks Tim, before opening an issue/FR to the documentation, ….
(maybe I misunderstood ... :-) )

i saw many questions about image import directory in the chatter…. that may mean that it needs to be explained deeper in he documentation :-)
Please correct me if I misunderstood, and consider if this could be used to improve documentation:

The first KEY point is that M2IF is a complete EXTERNAL solution to M2 (compliment again!)
This give M2IF a great speed upgrade over M2 in importing huge quantities of data.

But the DOWNSIDE of being external is that we will NOT find M2 doing anything we usually expect when we import data from admin backend.
Not even COPING image files in pub/media/catalog dir (!!), not even reindexing nor resizing the imported images, nor empting the cache !! (is it right ?)
The copy operation have to be configured by json file,
the reindex/resize/flush hast to be done by hand or by script.
(please Tim confirm ..)

But we could consider this “limitation” in an additional speed IMPROVEMENT ! (maybe this was the original idea?)
If we put DIRECTLY (and once) the images to be loaded in the pub/media/catalog/product/brand_x/model_y/type_z directory, M2IF will write our personal crafted path into M2 db path to base_image;
In this way the image files will stay there, as their final “official” path (and in a more logical directory structure), and we will NOT have to copy any files, sparing even more time (right?)

In the example above (images stay from start in final target directory), we will just put in the json config:

 “media-directory" : "pub/media/catalog/product"

and in the csv file, we will have the base_image as
/brand_x/model_y/type_z/img_01.jpg
We will also put:

"copy-images" : false,
 "clean-up-media-gallery" :false,

Because there is no need to copy nor to remove images...

IS it correct ?

Tim Wagner
@wagnert

@Giuliano69 You're right, we've to improve documentation, and for sure we'll do that during the next weeks. As mentioned in the Github issue, we're actually in a process of renaming M2IF to Pacemaker which takes some time. We're aware of the downsized you described above. So let me get a bit more into details ;-)

  • Not even COPING image files in pub/media/catalog dir (!!): M2IF is able to copy image files from "images-file-directory" to "media-directory" ( which have NO default values yet) if the flag "copy-images": true
  • The reindex/resize/flush hast to be done by hand or by a script: Yes absolutely (https://docs.m2if.com/38/getting-started/configuration#images), in general people who are using M2IF do that with a CRON script to make sure that everything has been reindexed, cleaned and resized

Pacemaker Enterprise exactly focuses an that tasks, so imagine it as a Pipeline (e. g. as you know it from Gitlab) that executes several steps whereas you have the possibility to define conditions, that defines when a step has to be executed and that prevents pipelines for steps to run concurrent, e. g. stop the indexer before the import starts. To your last question: Yes this is correct, you can read here how we handle it with Pacemaker Enterprise and how the directory structure has to look like (nearly the same as you described above what should work). But in short words, we try to avoid copy images during the import process, as this slows the import process down significantly and creates a lot of wast be renaming the files again and again.

Giuliano69
@Giuliano69

Thanks for the link about the image directory Path
AND the idea of symlinking instead of coping
(Maybe also mv command -inode relocation- will be equally fast...)

You warn about a hidden PROBLEM when want M2IF NOT to copy the image files:
because M2 will have to Find the path to the original image of the cache path
using the three level name regexpr |^.((?:/[^/]+){3})$|
we MUST use an * exact number of subdirs (=2)
lo finally store our image file.

The QUESTION is WHERE we should start counting the two subdirs ?
HERE I see some strange /difference between M2 and PeaceMaker/

Magento2
If we look at M2 standard image config for Luma test example :

<magento-install-dir>/pub/media/catalog/product/m/b/MB01-blue-0.jpg
<magento-install-dir>/pub/media/catalog/cache/0123456789/m/b/MB01-blue-0.jpg

Reading your doc and looking at Luma base, seems that we shoud start counting the 2 subdirs,
So we could place the image file in /product/ dir

<magento-install-dir>/pub/media/catalog/product/mydir1/mydir2/image.jpg

Here, starting from meda dir, we "cross" 4 dirs before the image.

PaceMaker
BUT, looking in Pacemaker dir tree the path is DIFFERENT

In PaceMaker, your layout for path is SKU/general/base_image.jpg, like

 /var/www/sftp/import/media/images/00490826/general/base_image.jpg

Considering that you symbolic linked

ln -s <magento-install-dir>/pub/media /var/www/sftp/import/media/images

your file is seen from the M2 point of view under

<magento-install-dir>/pub/media/00490826/general/base_image.jpg

(i.e PaceMaker "images" dir correspond to M2 "media" dir)

BUT ... you are missing the "product" dir in the "middle",
and starting from "media" dir, we cross only 3 dirs before the image.
You gave up to one dir !!

Does this this mean that product directory is NOT mandatory ?
By contrary: Can we use many "products" directory, to better organize images, i.e: ?

<magento-install-dir>/pub/media/catalog/product1/m/b/img1.jpg
<magento-install-dir>/pub/media/catalog/product2/m/b/img1.jpg
<magento-install-dir>/pub/media/catalog/product3/m/b/img1.jpg
Tim Wagner
@wagnert
@Giuliano69 the symlink should be ln -s /var/www/sftp/import/media/images <magento-install-dir>/pub/media/catalog/product which will result in something like <magento-install-dir>/pub/media/00490826/general/base_image.jpg. Directory structure pub/media/catalog/product is mandatory, as well as after that M2 at least needs two subfolders, e. g. m/b/24-MB01.jpg, which is a result of the method Media:: getOriginalImage().
Tim Wagner
@wagnert
@Giuliano69 Sorry ... I wrote a mistake :-( ... will result in something like <magento-install-dir>/pub/media/catalog/product/00490826/general/base_image.jpg :-)
Giuliano69
@Giuliano69

maybe the link info in
https://docs.met.tdintern.de/pacemaker/1.3/feature-configuration/import/import-media-files.html
are to be updated ?

giuliano@firenze:/var/www/sftp/import/media/images$ sudo ln -s /var/www/penstore/pub/media/ /var/www/sftp/import/media/images/
giuliano@firenze:/var/www/sftp/import/media/images$ ls
media
giuliano@firenze:/var/www/sftp/import/media/images$ ls -l media
lrwxrwxrwx 1 root root 28 mag 10 01:15 media -> /var/www/penstore/pub/media/
giuliano@firenze:/var/www/sftp/import/media/images$ cd media

then seems that sku dir 00490826 is before the 'catalog/product' path

Giuliano69
@Giuliano69
sorry, is better a screenshot of mc commander :-)
With that config it seemed that sku dir was complitely independent from the "usual" M2 dir...
Tim Wagner
@wagnert
@Giuliano69 I’ll check that tomorrow, hope this ok for you? 🙂
Tim Wagner
@wagnert
@Giuliano69 I've updated the documentation :-)
Giuliano69
@Giuliano69

Hi Tim
Thanks for your prompt update! this is very kind of you...

I would like to answer a question about product ATTRIBUTE UPDATE.
In [Usage] (https://docs.m2if.com/37/getting-started/usage) the shortcut add:update seems referred to apply only to product, being said
( New product data is added to the existing product data for the existing entries in the database. All fields except sku can be updated….. )

I tested it with
bin/import-cli-simple.phar import:attributes add-update
and it Works also for attribute !!!

I found that we can APPEND new values in the attribute list, even with the corresponding language labels.
BUT seems the add:update CANNOT update the labels for the attribute field in the storefront for other languages.

Is this intended or a working in progress feature ?
I enclose the two examples I made to add-update the basic attribute "manufacturer", in two steps.
The firs csv appended new attribute values in requested languages, AND updated the storefront label for said languages. (from Manufacturer to Brand , for all languages) The second csv appended the new attribute value (e.g. Pelikan) but FAILED to update the Italian manufacturer storefront label from "Brand" to "Marchio"

https://drive.google.com/file/d/1jOn-oBv-Ej1AIsm9vMPpaKUsHrlonO5-/view?usp=sharing
https://drive.google.com/file/d/1Doh8olGvHMt3xEfpQkyeHDCXyB1b_Sfm/view?usp=sharing

Should I open a Issue or a FR ?

Tim Wagner
@wagnert
Hi @Giuliano69 It would be awesome, if you could open an issue on Github, would that be possible? 🙂
Giuliano69
@Giuliano69
Thanks Tim, issue #249 created
Tim Wagner
@wagnert
@Giuliano69 Allright, I'll check it on tomorrow or on monday :)