For those not knowing what this is about
Disclaimer: I am no expert on this subject. This is just a general overview of different sources. Please correct me on anything that needs correction.
With pacman v5.2 Arch Linux has officially included support for a compression format developed by Facebook, Inc. called zstd. You can read an early proposal from March this year on the arch dev mailing list here and check the Github Repo. Arch is also planning to convert this to their standard compression method for packaging where formerly they had used xz instead which is based on LZMA.

This format is actively being developed by Yann Collet and Przemysław Skibiński at Facebook Inc. since February 2015. It is a multi threaded compression format in contrast to xz being single threaded. Arch developers have apparently decided that this would make for a way better compression ratio based on benchmarks which can be found especially in the mentioned mailing list.

From our end, this whole topic came up because of this thread.

Why this may raise concern in some people
Facebook is one of the so called "Big Four" companies and without a doubt one of the most powerful companies of this society we live in around the globe especially when it comes to private data. This is basically because we willingly tell Facebook everything about ourselves, but there's also a lot of code included in Facebook based services and servers that will try to get even more juice out of your not so obviously shared information. Basically things you never opted in to share. Like your actual physical location at any given moment, your telephone number, your interests and how much porn you really watch. Often this is data that can be collected because you use some service or another that is directly connected or owned by Facebook and you did not actively opt out of harvesting options activated by default. This is true not only for Facebook but for a lot of companies that make business with data, which are quite frankly the vast majority of all services you use on the internet.
Such companies are apparently also of high interest to governmental institutions such as so called intelligence agencies which are part of the military apparatus of a country which in return can usually provide vast amounts of money being (usually) the countries major investment. This is no news in itself and data harvesting companies in several countries (I suppose most) are even obligated to hand out information if it would assist in an investigation.

What's this gotta do with a compression format!?
Coming from one of those major companies the compression format in question is written in C and published under the BSD License and GPLv2 on Github. Connecting this with the former paragraph it is rather apparent that this might empower this specific company to become more and more a major part of the Linux community. Linux has grown and matured a lot and even gained user base on personal computers and it appears companies are getting aware of it. Also see Fungalnet's article on his blog about this where he mentions the recent purchases of Github and RedHat.
Being a part of something also means a certain influence and possibilities to exploit that something. This is true for our very life and any social construct to begin with. In this situation it may open way for Facebook to inject questionable code to harvest data into Linux. This is basically what this is all about.

My very personal opinion
Disclaimer: This is my personal opinion based on the little information I have on this matter. Please correct me where correction is due. Also I'm too lazy to search for sources here, as most of this is just very basic stuff.
To me personally the reason behind putting money and effort into the development of a multi threaded compression algorithm would suggest interest in pushing faster mobile and embedded device operation and gaining market share by being an active force in said development (see concerns and what that's gotta do with anything). AMD and Intel are fighting heavily for market share on PC/professional with immensely powerful work station and gaming CPUs getting smaller and smaller while ARM is already dominating the mobile market with just as powerful and more energy efficient processors for micro environments, which in reality is way bigger nowadays anyways.
For me there's several sides to this:
  • The zstd code is completely open source and could be checked for any questionable hooks or back doors at any moment. Although people might do that at first and later on get lazy and not get aware of such occurrence until it had already done some damage.
  • In a time where wireless radio frequencies are so fast that you can download a 10GB file in a few minutes (not where I live anyways) and processors that render almost life realistic video scenes in a matter of a few hours it seems absurd to opt for a compression gain in a package manager for desktop systems.
  • That being said Arch is not only desktop. It's also server, embedded and maybe even mobile in the future.
  • It is without a doubt a disruption for many Arch based distributions and Arch forks and implies decisions and work for their developers and maintainers
  • I've mentioned this in the causing thread but I'll repeat it here: Although I love to use, collaborate and share open source efforts and the general idea of working together to create great things I'm also aware that I'm looking at a (pretty old) LG monitor, listening to music from Spotify on (pretty old...) JBL stereo speakers using an Intel processor on an MSI motherboard with a SAMSUNG solid state drive. Hell even the enclosure all this resides in is not open source. It is made by Thermaltake. All these have in common that they make business, have business interests and make business decisions which in return empowers them to invest in doing more. I could not run Arch if it wasn't without any of this. Here's something to think about: There have been and still are open source solutions to processors, components, full systems like laptops and others yet most of them find themselves in a difficult situation. Plain business oriented companies can invest loads of money to advance and develop their stuff which is what everybody wants! If we stopped wanting more power, more screen real estate, more sound quality (I don't even see what for...), more speed etc. then we could start looking at alternatives. It is one major drawback of Linux/UNIX and all their derivatives that the relatively small developer base and private effort of a few without monetary compensation can not create the same in the same amount of time. (Although it can sometimes create even greater things in less amount of time). When it comes to sheer numbers, which our society claims for, quantity just wins over quality. No news here either. And if you're honest to yourself a lot of Linux development (as in evolution) happens because its major market is server and embedded and there is business decisions in this fact alone because companies want to use it for their benefit not for their spare time.
I feel like it is as with most things these days. If I do not want all this in my life I have to give up on a lot of things. Think about this because this last sentence is a harsh truth. I personally favor local organic food with a low carbon footprint. When I lived in Berlin, Germany this made me give up on most animal products which are insanely high in carbon/water footprint and made me become vegan at a point. I still am today but now I live in Mexico and it is insanely difficult to consume this way over here (as in most areas of the world). But it is on the other hand way easier to consume local food production which is just awesome. Although this is a far more complex topic it still relates to how much do you need or want to flee from our capitalist society, where do you want to draw your line and how far do you want to go? Is it necessary to become a lonesome warrior against "the system"? If it was for more warriors like us here on Obarun, would we need to fight "against the system" or rather should we look to "improve" it?

Discuss, share your views, your knowledge, your opinion!
( In a civilized manner apparently ;) )
https://lists.archlinux.org/pipermail/arch-dev-public/2019-March/029542.html

This is the test results arch is basing their decision on, I was inquiring to the resulting size of the pkg. From what I see this thing at high levels of compression can beat xz in size and time, but look at the memory it is using. So for a weak server building packages it can limit to how many it can do in parallel.

I hope building and installing packages can still be done through xz, because imagine with an old machine and low ram if you have to compile a kernel and compress it with zstd. The damn thing can rocket up to 8GB of memory, and the end result for a kernel might be 10% compression.

If fedora is using it why should arch fall behind the rush?
Mhmm I did try to compile a recent Kernel on my famous MacBook Pro from 13 years ago and to be honest it took the whole day to begin with :s
Not something anybody would want to do if not like absolutely necessary. But yeah you're right I didn't watch at the RAM there, that's quite high and would definitely not only take a day on my old machine now but actually blow it up :D A lot of current laptops and smaller desktops still use 4G pretty commonly even.

With the stand they're taking they just don't seem to care about outdated hard- or even software anymore, which was always a strength of Linux to begin with. I mean it's understandable to drop x32 if you don't have enough people to work on the maintenance, ask Eric XD But then again there's a pretty semi-official spin-off for x32, so what gives, I don't get it. And demanding high level resources of a community that takes pride in supporting almost any existent architecture and hardware is rather,...well...

By the way as I'm currently re-doing said MacBook I think it's hilarious how the "Kernel compression mode" option in the Kernel config says the following:
CONFIG_KERNEL_LZO:

Its compression ratio is the poorest among the choices. The kernel
size is about 10% bigger than gzip; however its speed
(both compression and decompression) is the fastest.
So it is the fastest bestest super duper mega format for compression and decompression, not so much for ratio. Let's see what LZ4 thinks about that statement:
CONFIG_KERNEL_LZ4:

LZ4 is an LZ77-type compressor with a fixed, byte-oriented encoding.
A preliminary version of LZ4 de/compression tool is available at
<https://code.google.com/p/lz4/>.

Its compression ratio is worse than LZO. The size of the kernel
is about 8% bigger than LZO. But the decompression speed is
faster than LZO.
Not only is it even worse in ratio, which LZO already claimed but it's also faster in decompression. All right! Hahahaha :D

Is Fedora also using zstd? that what you referring to?
pro:
compression is faster than any other compression format
decompression is faster that any other compression format
reproducible multithreading

cons:
increase in compressed package size
Increase in memory usage during compression
‘entreprise’ projects (linus as well if you consider the sponsor of linux foundation)
need an high level of resources (this is clearly a problem for a server or old machine)

as usual user will not take care about the things behing the hood and they just will see the gain of rapidity at installation phase like systemd with is awesome superpower boot.
The time of installation should be the last argument but on this world is the first one....
Arch do nothing just they follow Red Hat (as usual) then debian will follow and gentoo provide it support.

I'm not happy with their choice not about the 'Facebook' supports (the code is pretty simple to follow and to debug even for the HUF part) but about their philosophy which go far away once again from the Open Source projects.
But what about the choice for Obarun?
Jm and me have idea of packager but we don't have resources to deploy such a program and even less for building thousand of packages.
The next step for us will be the complete reframe of the FHS hierarchy, this will not happen on near future but all major distro will be merged in one...Systemd was the first step, compression format will be now applied,....

If linux represent the heaven i prefer the hell
Actually of all the boot sequences I have experienced by now, systemd is by far the worst and slowest...
where are those superpowers? ┐(￣ヘ￣;)┌
I think the difference has to be made between "Open Source" and "Non Profit". Apparently the late evolution of hard- and software has shown that open source can easily be profitable and even that profit can be based on open source.
Look at all the RISC-V buzz and the acquisitions happening in what once was more about non-profit than actually open-source. Open source was the standard, non-profit was the feeling, the love, the desire of a few. Although profitable projects have always been there on top of UNIX/LINUX no doubt.
But things have gotten big lately. Like seriously big.
Actually I just realized that Arch has GOOGLE_FIRMWARE included by default. And it even says:
Linux/x86 5.3.7 Kernel Configuration wrote These firmware drivers are used by Google's servers. They are
only useful if you are working directly on one of their
proprietary servers. If in doubt, say "N".
Wow!

Maybe adapt s6 for FreeBSD? lol T_T
Let's not be too negative by the way, I'd love to see Obarun stay strong and receive a lot of love and hopefully donators for Eric! Let's get Samsung on board for Eric! XD

  • [deleted]

  • Post #6
Does the community and me have to accept everything on the pretext that it is open source knowing the background of this company? You really do not see the problem, do you? Do I really need to make a comparison with the darkest period in our history?
Ask my grandfather and his neighborhood, during periods of big famine if they would accept an invitation to share an excellent and generous meal, cooked by their worst enemy knowing that the next day at the same table and by the actions of this same excellent cook there would have been empty chairs.
Do you understand the values I'm defending, and when the community have to say; no, we do not want it.
I am trying to cut a deal from the zapatista space program to be your sponsor, it is called the autonomous intergalactic space program

They will use Obarun in all of their spaceships and "ground control" :)
you make my day :lol:
What an awesome yet weird creative piece of art XD Is that actually something you own!? It's awesome.

Jean-Michel, I hope you're not targeting the wrong entity here, that is to say, your own user base. Most of us are here and actively sharing this experience with you because of that what you call "values" (I don't think we need to define these here). Every person individually defines "problems" and "values" though. The company in question and their persons in charge would probably call this an "opportunity" instead.
Also I personally think you can not compare Hitler burning people alive, with a company making efforts to contribute to Linux in hopes of market share, but well...to everybody his own. We are talking about computing. There would be no computing at all if huge companies wouldn't have invested billions of dollars in the transistor back in the days when that would have been like trillions of dollars today. I believe it's cynical to believe that computers are part of our society because of the good will of mankind to get connected and share. It's quite the opposite...
You can absolutely not build a CPU in your garage. We can stand up for the productive and positive usage of this technology though.
Please don't get confused, I "fought" for over a decade for the "values" you mention. I went to the streets weekly to show my nonconformity to my neighbors. You know what? I prefer powering off the computer and plant trees. Get some shit done. The computer has wasted half of my life for almost no gain in return. F*** this. Even living in rural Mexico planting trees I can't get off of this freaking drug. All the technological advance in compression algorithms and faster multi threaded cores is for people looking at screens laughing over other people hurting themselves. Good job mankind!

To return to the actual subject: I would also prefer to not have Arch make this their default format in their package manager. But it's dangerous and counter productive to get offensive between ourselves. If you wanted to open a thread on the Arch forums regarding this maybe other people might chime in and support that opinion, maybe not. I would. If you think it's not worth the effort, because Arch responsible staff will talk it down from the start, then you're hushing in front of a repressive dictatorship just like anybody else. It's hardly unlikely that Arch will change position on this, but it may be important to let them know that there's more than only one user concerned. If there is such a thread already, let me know, I'm not active on Arch forums. After all you're contributing to an operating system that is basically Arch, so it's a difficult situation in itself. Here's another metaphor: It's like wanting to borrow Hitlers type writers for their technological advance renting some punch cards blaming him for buying them from IBM and not building them in his garage. Something along those lines. Probably a little bit exaggerated but it may illustrate my point.

Again, I'd love to keep using Obarun and support you guys in your efforts. Let's keep things positive in a world full of bullshit.
No, not me :)
http://alfalfastudio.com/2019/07/16/ezlns-autonomous-intergalactic-space-program/
https://sfartenthusiast.com/2017/02/photo-feature-rigo-23-autonomous-intergalactic-space-program/

There is nothing personal in arts and sciences with zapatistas. They have had some global conferences on "comp-arte" and "con-sciencias" :) This is when experts meet and exchange ideas. The clash is what they have fun with, the one side comes and says I did this and that, the other side says we did this and we did that. They learn by teaching and teach by learning.
jean-michel wroteDo you understand the values I'm defending, and when the community have to say; no, we do not want it.
I fully understand and agree, but we can only do what we can do, voice our disagreement and disapproval. And it is important for others who disagree to know they are not alone. We are not organized enough to stop the tsunami of changes "they" decide, but it is a first step to know we are not alone, for when we decide to get serious about altering the course of history and our lives. We are not mature enough, and we fail to make the connection of our material problems with the overall problem, if we perceive it as a problem.

The hydra has many heads.

  • [deleted]

  • Post #12
@ marianarlt
I hope you're not targeting the wrong entity here, that is to say, your own user base.
I am not targeting anyone, I am speaking to everyone, and sharing my feeling, overall, to all the readers.
Also I personally think you can not compare Hitler [...]
I did not talk about him, no reason to reduce everything to that person.
with a company making efforts to contribute to Linux in hopes of market share
Does this company contributes to the well being of humans? What are the human values represented through this company ?
I "fought" for over a decade for the "values" you mention.
I was also a 'warrior' in my youth, today I have a different approach that is much more effective.
But it's dangerous and counter productive to get offensive between ourselves
No reason to feel offended. Precisely because we are between ourselves, we can talk, evaluate the situation and take other paths perhaps together.
Again, I'd love to keep using Obarun and support you guys in your efforts. Let's keep things positive in a world full of bullshit.
Thx, thx to eric, fungalnet, to you marianarlt and the others to keep things positive, for everyone and for yourself.
Well that person I mentioned is like the one fuckhead representing so much, so he's usually easy to refer to, sorry for picking him specifically :P
When I say "making efforts" it may sound as if I would value their work in some sort of way, because this specific verb has that secondary expression to it. It is apparently only a way of saying that they are working from their point of view on something that they think will be a net return for them. This is strictly from a business point of view.
Facebook, Google, Amazon, Bayer, Unilever and a handful of other, economically speaking strong and dominating, companies do not contribute to "the well being" of human society from my point of view. Theirs is probably very contrary to mine though. But then again I don't see how most individuals are being productive towards their own "well being" to begin with, even less so a well being of living together as people, something we call society. I know of very few people who consciously consume on a daily basis and I know even less who would actually read what the products they consume on a daily basis are made of and have a big red blinking, shouting alarm bell in their head, telling them that it's absurd. Internet as well is a consumer product. We consume it on a daily basis and often don't reflect on it. If we did we would start to turn away from some things and support others in favor. I'd argue it's mostly preferred to avoid socialization with valuable conversations where "we are between ourselves, we can talk, evaluate the situation and take other paths perhaps together". I like this sentence very much. It should be something prominent in our lives in general.

Thank you Jean-Michel.
> Internet as well is a consumer product

it is communication, specifically when talking about social media we are talking now about substituting physical communication with virtual.
Water is a "consumer product", food, shelter, access to land/water/forest is becoming a consumer product. The alternatives as to what you have to do to be able to consume are getting more and more "normalized". Not too far from what you had to do in a concentration camp. Some of you may be lucky to have escaped to small corners of the planet where this is not as brutal yet. The conformity to some channel of behavior that constructs your ability to "consume water".

But I will just leave it there, because I had made a promise to some people that I will keep my own choke hold on ... and it doesn't take for me too long to break through it :)

It is refreshing to see young people in Chile these days not giving up and accepting this dystopia as normal. You can't believe what hell it is to live in a place where people gave up.
2 months later
Since we all now have packages on our local repository /var/cache/pacman/pkg to compare with we can be witnesses of the superiority of this marvelous code/algorithm employed by the chairman of the arch for YOU!

https://sysdfree.wordpress.com/292

I should have broken the sequence and name the article 1984 instead of 292 but didn't think about it at that particular moment.

zstd is only faster when the size is much larger. When the size is smaller it is slower. The difference is infinitesimal.
Open source? I am not impressed. How many developers are capable of auditing a mathematical algorithm? The one has been around 3 years the other 40-50 years?

I dedicate this article to JM
The article link was placed in r/linux FIRST, then crossposted to archlinux and obarun and void (since I talk about void's relation with zstd).
After a group attacked me without any rational arguments or information, being crazy, being on drugs, THEN removed the article on the basis "of altering the title of the article and publishing something from a site "with offensive/intrusive advertising - wordpress.com". When I complained about the unjustified removal and explained it is my article I can change the title to whatever I want to, but the content is very much r/linux appropriate.

They banned me and one justified it as "reddit doesn't want you to publish your own but allows other to refer to your work", but they removed the article before I told them it was mine. :)

I feel very proud today ! For a long while as soon as I would post something there within seconds there were negative votes, which to me indicates bots running on the background downvoting something ahead of time so it doesn't appear on top of the category, it is shoved in the bottom.

THEY don't want too many people to know about this change and THEY will use this as precedence in the future to show support for the argument for zstandard. If I was participating in any distribution, in the future I would keep an eye on THE person that would propose to use it. Facebook has done many favors and has sent many people to jail and has tons of money to tunnel to the right "agents".
I've read it all now. Intense. Some reads are actually funny. Most of it is a pretty serious matter to me, hence why I made this thread to begin with. Thanks for the links and your blog in general Fungal. I conclude with JM: Congratulations on the ban, less disturbing and more positive time for you! Let's make things, not waste time on freakin' reddit :) Maybe just meditate whenever you felt like checking in on the r/gang ;) You're already contributing a lot, so every second for your own well being is a blessing for the whole community :)
what do you, this is my well being, I live for this stuff. :)
By the way, if you are around a multicore/thread machin try xz with the -Tn (where n is the number of threads allocated). The speed in negligible for pkg size files. The compression is better. The displayed tests show single thread performance for xz, very unfair. So why are they doing it anyway? Because facebook needs a large test base to develop it further and arch is usuing its userbase as guinea pigs. There is no other explanation. I've run tests over tests over tests, there is no logical support for this unless there is a "financial interest".

Powered by Obarun